In terms of chromosomes, as Monika has said, as soon as we are fertilised, we are either male or female, and in terms of chromosomes, that we stay (there are some odd peculiarities, like people who are XXY or XYY, or some other combination, but they are abnormalities, so I will ignore them in this context).
The problem is that chromosomes may carry the genes, and the genes may provide the blueprints for who we will be, but they do not build the human body itself - those genes must be translated into various enzymes, proteins, etc., and in this case, most crucially, into hormones.
In terms of hormones, we generally all start out as female, but if we have male hormones, we will change to being male. The default state for an animal (certainly for a mammal) is assumed to be female.
The point is, at what stage do these hormones begin to act. Certainly, there are hormones that act within the womb, including sex hormones (and even females have a small amount of male sex hormones, just less than males have). When a baby is first born, you can sex it from day one, so clearly there must have been hormones within the womb that caused some sexual changes to occur long before then (but I doubt those hormones were active in the very earliest stages of pregnancy - since right at the beginning, it is not even easy to tell whether you are looking at the first stages of a human or a chicken, let alone what sex it is). On the other hand, we also know that hormone levels continue to rise until puberty, when the final aspects of sexual differentiation take place.