Hi Aeris and everyone.
I needed a distraction so I had started writing something for this thread a few days ago but it has grown a bit too large. Rather than just throwing it all away, I might as well just post what I've done. It's long, so feel free to just ignore it. I've not checked the spelling or grammar or tidied it up much, sorry
I have a bit of time to look through this and add some comments, I hope you won't mind.
Let's race through what we use as the basis for this sort of cosmology and see where "dark energy" and the expansion of space apppears and begs for some explanation. This is General Relativity and especially the Friedmann equations as far as I'm concerned.
Space is too complicated to construct the actual stress-energy-momentum tensor that should be used in the Einstein Field Equations. There are too many planets, bits of gas and assorted stuff all over the place. We find new meteors in or near our solar system all the time and looking ouside of our solar system we find new stars with our telescopes every week. The recent Hubble "deep field" photographs demonstrate that what may appear to be fairly empty space at first glance is likely to be full of stars and indeed whole galaxies.
So let's assert that there is no hope of constructing the actual stress-energy tensor we would really want. Some sources will mention that we couldn't realistically solve the Einstein Field Equations with such a complicated stress-energy tensor anyway - this is true but almost irrelevant. It's not a question of how much computing power could be brought to bear on the problem. We just DO NOT HAVE the information we need - we would need to know the location of everything in the universe.
So, what we do is simplify the situation. There is an assumption that the universe is isotropic and homogeneous. The commonly used models treat space as if it is filled by something that is referred to as a "cosmological fluid". What is a cosmological fluid? See textbooks for a full answer. I've got to rush over things here: A cosmological fluid is directly comparable to an ideal fluid in other areas of physics. Our comsological fluid has many components, each component would behave like a fluid in it's own right. The cosmological fluid is the fluid you get when you mix all those components up perfectly ("stir well") to create a homogenised fluid. The use of the term "fluid" isn't suggesting that there is some liquid thing in space, just that the stress-energy tensor we need for GR is equivalent to the stress-energy tensor produced by an ideal fluid.
- - - - - -
First stop, or pause for thought
What if modeling the contents of space as if it is a cosmological fluid is NOT reasonable. Specifically, we know that space is not isotropic and homogeneous at small scales. What if this makes an important difference to the final results you extract from General Relativity and the Einstein Field Equations (EFE)? Let's phrase this another way: We cannot use the correct stress-energy tensor in the EFE, so instead we use something that is like an idealised average and assumed to remain quite unform across the universe. We simply hope that the results we obtain will apply to the universe in some average sense, or over large enough scales.
This might sound like a small issue, if you input an average value n times then you get an output which should be much the same as if you had really inputted n different individual values. This works well where the function performed on the input is "linear", but it falls apart if the function is non-linear. The sort of calculations and manipulations perfomed in GR are sometimes non-linear.
- - - - -
Anyway, what do we get from the Einstein Field Equations with this assumption? We get the Friedmann equations.
Here's the first Friedmann equation:
= 
[This form of the Friedmann equation is from page 338, An introduction to Spacetime and Geometry, Sean Carroll]
where H = The Hubble "constant" (but it's important to note that H= H(t) will vary with time).
G = usual Gravitational constant.
ρi = energy density of the i th component of the cosmological fluid. There is just one value of the index i which represents a "fictitious" energy density contribution from the curvature of space. To say that another way, there is a term ρc which is NOT the energy density of a component of the cosmological fluid, it is just here to replace the need to add a term dependant on k into the equation. (Just for the sake of completion, ρc =
where k = curvature. So that, ρc, the energy density due to curvature is positive if k is negative and vice versa).
What do we see from this equation?
H is real valued, so H2 is certainly non-negative.
Therefore, it is not possible for all of the energy densitiy components ρi to be negative.
We can apply the model for arbitrary cosmological fluids, for example, where there is JUST matter and no other fluid components. There could always be a fictitious energy density for the curvature of space, which is a shame - but we can at least reduce the right hand side of the equation to a sum of just two components. At least one of these must be non-negative by the earlier reasoning.
So if the energy density contribution from curvature is negative then the energy density of any (proper) component in the cosmological fluid must be positive. Editing: This post is too long, I can't go through all the arguments that suggest energy densities should be positive (they are only suggestions anyway).
We could just make the reasonable assumption that the energy density contribution of any real component of the contents of space must be positive. I mean, this is quite reasonable, if there is something there then it should have some mass (which is equivalent to energy density) or else be something with a positive energy like a photon. However, the full situation is potentially more complicated and I would refer anyone to chapter 4, section 6 of the aforementioned book (Spacetime and Geom, Carroll) where the various Energy conditions that might constrain the stress-energy tensor are discussed. Without further discussion, we will be assuming the Weak Energy Condition (WEC), so that all energy densities of any component of our cosmological fluid will be non-negative.
OK, We need to take the idea that ρi ≥ 0 for every real component of the cosmological fluid and go forward to look at the second Friedmann equation:
= 
Where ρi = energy density of i th component (as before) ;
Pi = Pressure of the i th component.
and a = a(t) is the scale factor.
In this equation we don't have to include the index, i, that represented a component from the curvature of space. We can if you want but the (fictitious) pressure from the curvature component would be precisely negative one-third of it's fictitous energy density, so the whole term ρ + 3P becomes 0 and including another 0 in the sum is just needless work.
Now we are assuming that every ρi ≥ 0. So if the pressure, Pi , from each component of the cosmological fluid was also positive, then the entire right hand side of the equation is negative. Since the scale factor, a, is always non-negative, we see that
would always be negative.
This quantity
is one parameter that can be used to describe the acceleration of expansion of space. There is another parameter
which can also be used to examine and describe the acceleration of expansion. They are not the same and there can be situations where
> 0 but
<0 (for example consider a(t) = t2). This, I think, is where most of the discrepency between Halc and Origin has come from in one of the earlier posts.
I'm going to follow the convention from the textbook previously mentioned:
In practice, "accelerating" usually refers to a situation where
>0, even if
< 0.
[page 339, Spacetime and Geometry, Sean Carroll]
Anyway, what we conclude is that if every component of the cosmological fluid had the properties you would imagine for a fluid - specifically that they have positive energy density and positive pressure - then it is inescapable that the acceleration of the expansion of space would be negative (
<0). To paraphrase this, the expansion would NOT be accelerating it would be slowing down.
This is not what we have actually observed. In our real universe it seems that
> 0 at the current time ("the expansion of space is accelerating"). Let's do exactly what some physicist(s) had to do: Look at the Friedmann equations again and recognise that what we need is a component of the cosmological fluid that is most unusual. It would have properties that are unlike any real fluid you might find in your house. We need a component with negative energy density OR ELSE a negative pressure (or both would be just fine). This really strange component of the contents of space is what you might like to call "dark energy".
1. Could it be something with a negative energy density? Well, maybe. This would violate the Weak Energy Condition (WEC) but maybe that was just too restrictive and far too optimistic anyway. We could replace this with another condition like the Null Energy Condition (NEC) which effectively allows a cosmological fluid component to have negative energy density provded the pressure of the fluid is positive and compensates sufficiently. What does this mean? Let's be honest, it's not worth discussing here. See page 175 of Spacetime and Geometry, Sean Carroll if you're intersted. It's not the choice that most physicists have made and seems less promising as an explanation of what dark energy is.
2. Could it be something with a negative pressure? Again maybe. This is the choice more physicists will make. It is possible to imagine that there is a component of the contents of space that we will call "vacuum energy". Vacuum energy can be modeled with the stress energy tensor of a fluid with parameter ω = -1. All this means is that it's a fluid (as far as the stress energy tensor is concerned) and its pressure is precisely the negative of its energy density.
3. Just for the sake of completion we could mention that if we were to assume the Strong Energy Condition (SEC) must apply to components of the cosmological fluid, then it is inescapable that ρ + 3P ≥ 0 for every component and then
is always negative. Hence, the acceleration of expansion is quite bad news for the SEC: At least one component of the cosmological fluid must have properties that violate the Strong Energy Condition.
Are we all done? Almost, we need to just take a quick glance at the Einstein Field Equation (EFE). Here it is:
Gμν = k . Tμν (where k = a constant)
The left hand side involves the geometry, especially the curvature, of space. The right hand side involves the contents of space. In particular, the right hand side involves the energy and momentum of the contents of space. It's an equation, so if you change one side then the other side must change and vice versa - change the other side and the first side has to change. There is a well known saying that I will paraphrase here: The left hand side tells matter how it should move. The right hand side tells space how it should curve.
At the moment, with the EFE in the form shown above, vacuum energy is included in the over-all stress-energy tensor of the universe Tμν but we can break this tensor apart into two pieces, one part from the vacuum energy and the other part being... well everything else that isn't vacuum energy (so that will be the other cosomological fluid components we had in our cosmological model).
We get this:
Gμν = k . T'μν - Λ. gμν (where k and Λ = some constants). Note that T'μν does have a dash ' in it but it's not printing clearly - it's not quite the same as Tμν we have subtracted the vacuum energy component out of it.
Anyway, being an equation, we can take the last term over to the other side as usual. We get this:
Gμν + Λ gμν = k . T'μν (where k and Λ = some constants)
Now that form of the EFE shown just above is known as the EFE with a cosmological constant, Λ. It describes how space curves in response to some stress-energy given by the tensor T' under the assumption that there should be a cosmological constant Λ. Equally the equation tells us how the contents described by the tensor T' must move in response to spatial curvature given by the left hand side (under the assumption that there is a cosmological constant Λ).
There is very subtle shift here but it all happens just because something was put on one side of the equation and not the other side. I'm going to phrase it another way: With a cosmological constant, there is a change in the rules of movement for the contents of space. However, the contents of space is given by the tensor T' which does not need to include any component so strange as vacuum energy (we subtracted that bit out).
Without a cosmological constant the rules of movement are different (arguably simpler) BUT there must be a strange component in the contents space that behaves like vacuum energy.
Finally we're ready to answer a question from Aeris directly:
Has anyone ever considered the possibility that Dark Energy may not exist at all and that the space can just expand without any outside force dark energy acting on it?
Yes, very much so. With a cosmological constant in the Einstein Field Equations, the rules of movement for the contents of space are just different and space can expand and indeed accelerate even if all it has in it is just the more conventional contents (not dark energy) that we're all happy with.
Is that all there is to it then? There isn't any dark energy, it's just that we should have a cosmological constant in the EFE, i.e. just accept that General Relativity needed a slight adjustment?
Most of what we attribute to dark energy can be explained by a non-zero cosmological constant but not necessarily all of it. The terms "vacuum energy" and "dark energy" are sometimes used interchangeably but there should really be a fine distinction between the two. Vacuum energy is presumed to be a constant and uniform characteristic (or component) of any volume of space, so it's energy density never changes. No matter where or when you sample 1 cubic metre of space there should be the same amount of vacuum energy in it. Dark energy may be something that does have a little variation through space (and possibly time).
Here's one example: Refer to page 360 of Spacetime and Geometry, Sean Carroll - where it is suggested that the comsological constant may not quite as "constant" as we might have hoped. This means it would be more like a parameter or quantity associated with something specific to that piece of space (let's call it the "dark energy" in that bit of space). So dark energy mimics the behaviour of vacuum energy as far as General Relativity is concerned but the amount of dark energy might vary slightly across the universe and through time.
The spatial variation in dark energy should be small (according to Carroll) otherwise the effects of a high concentration in one area of space would have been noticed similar to the way dark matter was detected.
--- This is too long, I'm stopping now. Bye and best wishes ----