[AllenG (OP)]

In humans that is.

I remember being in the play pin with Paul (my brother).
I can remember not being able to walk and pulling up on the couch.
I remember having my diapers changed. I clearly remember the pins.
I remember the mobile over my crib from the point of view of looking up at it.
And I have many others.  Although they are mostly flashes of memory, I can pick out distinct details.

My siblings insist that I'm a freak, but do collaborate what I remember.   

When does permanent memory normally start to form?

[Chemistry4me]

How exactly does memory occur? After years of study, there is much support for the idea that memory involves a persistent change in the connection between neurons. In animal studies, scientists found that this occurs in the short term through two biochemical events that affect the strength of the relevant synapses. The stability of long-term memory is conferred by turning on genes that may lead to modifications within neurons that change the strength and number of synapses.

[Chemistry4me]

Brain cells also form too many connections at first. For example, in primates, the projections from the two eyes to the brain initially overlap and then sort out to separate territories devoted only to one eye or the other. Furthermore, in the young primate cerebral cortex, the connections between neurons are greater in number and twice as dense as those in an adult primate. Communication between neurons with chemical and electrical signals is necessary to weed out the connections. The connections that are active and generating electrical currents survive, whereas those with little or no activity are lost. Thus, the circuits of the adult brain are formed, at least in part, by sculpting away incorrect connections to leave only the correct ones.

[Chemistry4me]

The brain’s refining and building of the network in mammals, including humans, continues after birth. An organism’s interactions with its surroundings fine-tune connections. Changes occur during critical periods. These are windows of time during development when the nervous system must obtain certain critical experiences, such as sensory, movement, or emotional input, to develop properly. After a critical period, connections diminish in number and are less subject to change, but the ones that remain are stronger, more reliable, and more precise. Injury or sensory or social deprivation occurring at a certain stage of postnatal life may affect one aspect of development, where as the same injury at a different period may affect another aspect.