[Eternal Student (OP)]

Hi.

    I'll assume you're familiar with the Shell theorem.    https://en.wikipedia.org/wiki/Shell_theorem

    Now supppose you're considering a simple model of star formation in the early universe.   We'll have a perfectly uniform distribution of Hydrogen gas spread out over space  AND  we'll also assume that space is infinite.   [NOTE:  It is not necessary or guaranteed that the universe is or was infinite,  that doesn't matter, for this hypothetical situation we are having an infinite space].    For a star to form, some hydrogen has to clump together under gravity and form an overdense region of Hydgrogen.

     We can choose to put an origin somewhere and draw a sphere of radius r around that.   Now, consider a particle of Hydrogen on the boundary (surface) of that sphere.   By the shell theorem, we can see that the Hydrogen particle is attracted to the origin by gravity.  We also see that we can ignore the attraction to anything outside that sphere of radius r.   Why?   We can draw hollow shells of radius R > r and the shell theorem tells us that since the Hydrogen particle is inside those spheres, it expereinces no net force due to those larger shells.   All of space is just a collection of hollow spheres of radius R > r    plus the solid sphere of radius r,   so we have considered all of space.     So there is just a net force toward the origin and we can even calculate this if we wish.        F =   GMm / r²       with m = Hydrogen particle mass,     M is the Mass contained in the sphere of radius r, so M = 4/3.ρ.π.r³   for some density ρ which we assume to be uniform across space atleast until some movement of Hydrogen particles to one place does start to happen.

   Now, of course, the puzzling thing is that we chose to put the origin somewhere and we could just as easily have put the origin somewhere else.   Measure the distance from the new origin to the same Hydrogen particle, that is your new value of little r you will use.  We could repeat the argument with the shell theorem (the particle is on the surface of a sphere of radius new r from the new origin etc.)  We obtain the resut that the same Hydrogen particle is no longer being pulled toward the old origin, it is only being pulled toward the new origin location we have now selected.
    Obviously, the Hydrogen particle cannot have a net force in TWO different directions.   It can't be accelerated  ---> that way and also   <---- that way  and also   /  this way,   \ this way  and every other way you can imagine.

    Which thing went wrong?   Which bit of the shell theorem broke down and why?

Best Wishes.

[Bored chemist]

Which bit of the shell theorem broke down and why?

The assumption of homogeneity
Because stuff is made of atoms and molecules.

[alancalverd]

BC beat me to it by 30 minutes (I was asleep in the garden).

[Halc]

I disagree with the prior answers. The space is homogeneous for the purpose of this connundrum. Going down to the particle level is never part of the shell theorem, which technically fails since no collection of discreet masses can be spherically symmetric.

We'll have a perfectly uniform distribution of Hydrogen gas spread out over space  AND  we'll also assume that space is infinite.[NOTE:  It is not necessary or guaranteed that the universe is or was infinite,  that doesn't matter, for this hypothetical situation we are having an infinite space].    For a star to form, some hydrogen has to clump together under gravity and form an overdense region of Hydgrogen.

Which kind of departs from your premise of a perfectly uniform distribution. Yes, anomalies form early on and escalate into more dense regions. It is a stretch to suppose that they are spherically symmetrical anomolies.

We can choose to put an origin somewhere and draw a sphere of radius r around that.   Now, consider a particle of Hydrogen on the boundary (surface) of that sphere.   By the shell theorem, we can see that the Hydrogen particle is attracted to the origin by gravity.

Sure, but it is alto attracted in all other directions equally since there is a similar sphere at which it is on the edge, in any direction. The particle doesn't accelerate since the net force is zero. Your application of the theorem is obviously invalid, and your logic stands up, so the theorem cannot apply to an unbounded uniform distribution of mass since it doesn't satisfy the definition of a spherical distribution.
It is the selection of the origin that is the invalid step. That origin needs to be put at the center of mass of the universe, and an infinite universe doesn't have one.

[Bored chemist]

Now, consider a particle of Hydrogen

The space is homogeneous for the purpose of this connundrum

Pick one.

[Halc]

Now, consider a particle of Hydrogen

The space is homogeneous for the purpose of this connundrum

Pick one.

The particle is the test mass being attracted to the homogeneous everything-else by a net force of zero. It doesn't even need a specific mass since it is the coordinate acceleration at its location that we're after, the gradient of the gravitational field there.

[alancalverd]

But by definition is isn't locally homgeneous. All the particles are moving at random, so the particle density at any point will vary over time from less than the mean to more than the mean, and if D_loc > D_mean you can get gravitational agglomeration.

[Eternal Student (OP)]

Hi.

   Yes, you could try and explain why a real world situation wouldn't be like the situation in the model.
   We could counter those arguments by suggesting we have photons instead of Hydrogen particles.   So, wind the universe back a bit further, there is just infinite space and a uniform density of e-m radiation in it.   Now photons don't occupy a definite location or have particulate properties exactly like Hydrogen would.   However, they do have energy and so they do gravitate (act as a source of gravity).
     It's a bit debateable in that we would need GR to determine this gravity and it may not be that we obtain something like the  1/r²  we have for Newtonian gravity.   We do need a 1/r² relationship for the shell theorem to hold.
      None-the-less, I'm just going to hold my ground.   This is a hypothetical situation, we assume that there is something that is adequately described by a uniform density through space, ρ, it is not necessarily particulate so that we can only have  "some here, none next to it, some here here etc."  - there is precisely   ρdV   of it  in any volume element dV.   None-the-less a volume  dV  of it will be assumed to experience a change of momentum and move as directed by Newton  with mass, m = ρdV.   (Just to get ahead and address some of the issues raised later..... the distribution may not continue to be uniform at some later time > initial time, i.e. if some clumping of the Hydrogen does happen.   We only need to consider the initial action on a test particle given the assumed uniform initial distribution of mass across an infinite space.    After all, if there is no initial attraction of the particle to one place, then the situation does not change at  initial time + δt.   Even if you want to assign every particle some random initial velocity, there's no force on any particle at initial time, so it's velocity does not change and the density remains uniform if particles do move randomly..... So  the picture at time = initial time + δt is identical to the picture at  time = initial time).
   

The space is homogeneous for the purpose of this connundrum.

    Thank you @Halc  ,  that is the essence of it.   This is a hypothetical situation and there is no need to tarnish it by suggesting the real world isn't as perfect as the model.

I'm still reading the later replies but need some time to do some things.

Best Wishes.

[Halc]

Yes, you could try and explain why a real world situation wouldn't be like the situation in the model.

Well, at some point you have to explain how matter started to clump rather than remain perfectly homogeneous. For that, you have to go back to the earliest epochs (long before there were protons and such) when there was fantastic energy density, and then apply quantum fluctuations to the situation. Lacking a unified field theory, this is all still pretty speculative. One also has to go about explaining the annoying matter/antimatter imbalance.

We do need a 1/r² relationship for the shell theorem to hold.

that's right, and 1/r² doesn't hold up for extreme cases (like near neutron stars and such). But the shell theorem is totally inapplicable here, and by the simplest symmetry, there should be zero net forces on anything except due to those local variances which come only out of quantum theory, so GR isn't going to be a huge help over just doing this the Newtonian way.

I'm not sure if this topic is about the applicability of the shell theorem to an infinite homogeneous mass distribution (answer: not applicable), or if it is about how matter started to clump (need a different model to guide us) early in the game.

We only need to consider the initial action on a test particle given the assumed uniform initial distribution of mass across an infinite space.

This suggests the former of the two.

Even if you want to assign every particle some random initial velocity, there's no force on any particle at initial time, so it's velocity does not change and the density remains uniform if particles do move randomly.

Except in an expanding metric, any initial velocity drops off almost instantly, so in pretty much no time, nothing has any velocity except due to local accelerations from local interactions. Kinetic energy is not conserved under non-inertial metrics, which includes expanding metrics.
So essentially, the entire universe is full of stationary 'stuff' (at least once 'stuff' starts to form). OK, radiation continues at light speed, but its energy similarly drops off really quickly, so a photon, if it manages to get anywhere in an opaque universe, will in short order increase its wavelength.

So the picture at time = initial time + δt is identical to the picture at  time = initial time).

Heck no. The density went way down over that time, but sans local fluctuations, the place is still nice and uniform energy density and there should be no net 'forces' acting on anything.

This is a hypothetical situation and there is no need to tarnish it by suggesting the real world isn't as perfect as the model.

Well, it is those imperfections that eventually brought about galaxies and such. Without them, the universe would be a permanent gray paste. That's why I asked if this topic was about the actual homogeneous case, or if it was about how the clumping starts.

[Halc]

Sorry for all the posts, but thought about this some more, and want to avoid stating things that sound unconditional.

We can choose to put an origin somewhere and draw a sphere of radius r around that.   Now, consider a particle of Hydrogen on the boundary (surface) of that sphere.   By the shell theorem, we can see that the Hydrogen particle is attracted to the origin by gravity.  We also see that we can ignore the attraction to anything outside that sphere of radius r.

Back to the OP argument, since one can 'frame' my rebuttal in a way that refutes my rebuttal.

I think that under Newtonian law, the sort of logic that you spelled out above is valid in an inertial (as opposed to expanding) frame, and that our test particle does indeed acceleration towards any arbitrary origin that you select. Hence, lacking any recession rate of that arbitrarily selected origin, all that homogeneous matter will collapse in due time.  This is barring wrenches in the works like dark energy which isn't compatible with Newton's theories anyway.

So how does switching from Newton to GR affect that? Well, the shell theorem certainly has trouble under GR. Under GR, there is no possibility of a uniform gravitational field, and yet the shell theorem can show that an off-center spherical hollow in an otherwise solid homogeneous spherical mass will (like an infinite sheet of mass) produce a uniform gravitational field. Since GR doesn't allow this, the shell theorem doesn't work in general, and only approximates a solution for low mass densities.

GR of course also doesn't allow a uniform distribution of mass in an inertial frame, so any treatment using such a frame is wrong right out of the gate.

[alancalverd]

We'll have a perfectly uniform distribution of Hydrogen gas spread out over space  AND  we'll also assume that space is infinite.   [NOTE:  It is not necessary or guaranteed that the universe is or was infinite,  that doesn't matter, for this hypothetical situation we are having an infinite space].    For a star to form, some hydrogen has to clump together under gravity and form an overdense region of Hydgrogen.

Eppur si muove (Galileo)

[Eternal Student (OP)]

Hi.

    There is a lot of new replies and it may be easier to take them in a different order to the time at which they appeared.

I'm not sure if this topic is about the applicability of the shell theorem to an infinite homogeneous mass distribution (answer: not applicable), or if it is about how matter started to clump (need a different model to guide us) early in the game.

    I was primarily thiking about the first thing:   The applicability of the shell theorem to an infinite expanse of space with a homogenous mass distribution.
    I have kept the second thing:   How or why matter clumped in the early universe in the background.   I've mentioned it just so that the problem isn't immediately classified as purely abstract or theoretical with no application.    Anyway, I've no objection to discussing the Cosmological issue and probably would have done so eventually.   However, you ( @Halc ) have probably already said enough on that.

   We seem to be in reasonable agreement about how the universe may have actually evolved and can dispense with @alancalverd 's statement   "epuur si muove".
    It is quite likely that there were always some density differences in the universe.   This could be as minor as quantum fluctations exactly as you stated.    Starting with just small density differences in different places and running simulations of gravity and Newtonian mechanics (N body simulations) we can get larger structures (like stars, solar systems, galaxies and, of course, voids in between those things) to form.   
    If the universe had been perfectly smooth or homogeneous at some early time, then it seems / is conjectured that Astronomical structures of the type we know would never have formed.

   Anyway, that's the current Cosmological ideas.   That's why it seemed worth taking a moment to really ask why an infinite expanse of homogenous material wouldn't start to form over-dense regions somewhere.   Hence the OP.

      What's under this spoiler is just another statement of the OP, you can skip it.  I know this post is too long.

Spoiler: show

   Suppose we have an infinite space with an initially uniform density of material, ρ, everywhere.   Consider a test particle located somewhere in that space,  at position  x  .   What force would that test particle experience?

   We can answer that by using symmetry:    There can't be a net force in direction  --->   because there is nothing special or different about the direction  --->  rather than any other direction.   So there must be 0 net force, that's the only way you can have it.   That obtains the usual or expected outcome then,  a bit of Hydrogen in this space is never pulled anywhere and no clumping at a point should ever happen.   If the Hydrogen was at rest initially  (which as you correctly point out for an expanding universe is actually a very reasonable assumption as the scale factor increases) then it just sits there and stays at rest for ever more.
    However, applying the shell theorem (as it was done in the OP) can get a different answer.   The Hydrogen can be shown to have a net force in the direction of the origin.   The origin can also be wherever you want.   This is seemingly a bit of a contradiction, so you've got to ask -   what bit of the shell theorem, or assumption went wrong?

    Now I can address some of the earlier points you ( @Halc ) raised about the Shell theorem.....

You can also skip this spoiler.

Spoiler: show

 

Sure, but it is alto attracted in all other directions equally since there is a similar sphere at which it is on the edge, in any direction. The particle doesn't accelerate since the net force is zero.

    You seem to agree that the shell theorem would show a non-zero net force exists in some direction.   You also seem to agree that direction is arbitrary.    The last sentence doesn't need to follow though.     You don't apply the Shell theorem with all possible origins and then sum all predicted net forces to obtain the actual real and very final net force.    You just apply the shell theorem once and provided you have included all of space (and hence all matter) in the outer shell(s), then you are done, you have the final net force on the test particle.    So the conclusion is not that there is no net force, it's that there is a net force but it seems to be in any arbitrary direction.   Worse that that, even it's magnitude is arbitrary  (just move the origin further away but in the same direction,   the inner sphere grows ~ r³ in volume and hence in mass,  while the gravitational force from it falls off only as ~1/r²,   making the total force vary ~ r  = the distance from our test particle to the origin).

   You (Halc) sometimes make arguments that the space isn't suitably spherically symmetric....

...the theorem cannot apply to an unbounded uniform distribution of mass since it doesn't satisfy the definition of a spherical distribution.

     For an infinite space with uniform density everything is spherically symmetric about every point in that space.
Pick a point in the space and consider what exists at some distance L from it and at an angle  θ.    Now compare that with what exists at the same distance L but in the direction at an angle Φ.    All that exists in the space is the stuff we have stipulated will have a constant density, ρ, everywhere.   There are no points in space where a lack of spherical symmetry is exhibited.

    The only part I can't dismiss is that the shell theorem may not apply to a shell that is unbounded.   However, it is used in Astronomy often and assumed to apply across (all of) space which isn't precluded from being unbounded.   Example, trajectories of stars in a galaxy can be estimated by considering only that mass which is in a spherical region interior to that star etc.

   Anyway, just to be clear, what you're suggesting is the following:
A particle in the interior of a shell experiences no force due to gravity from the material of that shell.   That holds for shells of some finite thickness (assuming the usual things like spherical symmetry).   However, when that shell is so thick that it's of unbounded extent, the result may not hold, the particle might experience a force.

    I've not seen any statement of the shell theorem that specifically limits you to an outer shell of finite size.   Just to check, let's derive the Shell theorem from first principles (the quick way) and check that we really didn't care about how thick the outer shell may be.
    We'll take Gauss' law as our starting point because that's quicker than going with Newton's original formulation of gravity.
    {Recall that LaTeX is not working, sorry}    So just assume I've done something similar to this Wikipedia section:
https://en.wikipedia.org/wiki/Shell_theorem#Derivation_using_Gauss's_law
    Notice that the only assumption we required was that the mass distribution is spherically symmetric about the centre of our inner sphere of radius r.    The thickness of the outer shell was irrelevant because that mass was never enclosed by our surface and never contributed any net flux in the flux integral over that surface.  We never needed to know anything about the outer shell material, it wasn't (for example) assumed to have density -->0 as  the radius ---> infinity or something like that.   All we needed is that the assumption of spherical symmetry is satisfied so that we could make the argument that  g ,  the gravitational field at the surface of the inner sphere, couldn't have been to the left or right, it had to be radially orientated  and also due to this symmetry it's magnitude depends only on r = |r|  and not on the angle θ.

Obviously this is too long and I'm signing off.

Best Wishes.

[alancalverd]

This seems to come down to the question of whether an infinite homogeneous medium can be considered spherically symmetric. Not "from the inside" because of course there is no "outside" if it is infinite.

Rather than wander off into the barren land of philosophy, we can say that anywhere close to our origin is indeed spherically symmetric because it is homogeneous, and anywhere further away becomes increasingly less significant as we increase r, so as r → ∞  the effect of regions > r cannot become less symmetric. 

[Eternal Student (OP)]

Hi.

    I've just been Google-ing for more answers.

   Much the same question was asked here:

https://physics.stackexchange.com/questions/490829/ambiguity-in-applying-newtons-shell-theorem-in-an-infinite-homogeneous-universe

On the positive side, the original post in that website has identified much the same issues.  It has a nice diagram and the basic situation is described clearly.

On the negative side,  I don't especially agree with the highest ranked answer.   Their response pivots around the inability to identify a unique gravitational potential function, Φ, in such a space.    However, it seems possible to derive the shell theorem using the form of Gauss's law for gravitation which never involved defining or worrying about whether a gravitational potential function exists or would be unique  (see earlier post here, which linked to a Wikipedia proof).
   Briefly,  where a potential function Φ(x) exists then we have   g (x) =  ∇Φ    but there is no reason why a well defined gravitational field g can't exist throughout space even if a suitable potential function Φ could not be found.    Gauss' law is not    ρ =  -∇² Φ   as stated in the stack-exchange reply.     Gauss' law is  (well I don't have the ability to use LaTeX for mathematical notation) - the statement that a flux integral involving only g is equal to a volume integral of the mass density enclosed   (or the differential form of that if you prefer).   Where you do have a suitable potential function it's trivial to show that the stack exchange version of Gauss' law is equivalent to the version I have stated.   However, defining or constructing a potential function is a side-route that you do not need to take, there is no demand for a well defined potential function to exist, Gauss' law as stated (a volume integral = a flux integral) never required it.
    The other answers vary in quality and relevance.   There is one about some history and how major figures like Newton may also have noticed this problem that the shell theorem produces in this situation.

Best Wishes.

[Halc]

Obviously this is too long

I tend to do that as well, but I'll try to break up my reply, which is slow in coming since I've been kinda busy.

Dealing with your first spoiler:

Suppose we have an infinite space with an initially uniform density of material, ρ, everywhere.   Consider a test particle located somewhere in that space,  at position  x  .   What force would that test particle experience?

First of all, the sorts of forces you describe cannot be 'experienced', which is why an accelerometer reads zero despite its coordinate acceleration in orbit, and it is very much coordinate acceleration you're after here, not proper, which happens due to something usually contacting you on one side and deforming you. So the acceleration of the particle seems coordinate system dependent, thus the question of if it is really accelerating towards that arbitrary CoM you defined, or the stuff at that origin is accelerating towards the particle, is an abstract question, not a physical one.

Also, in your initial homogeneous state, is everything stationary in some inertial frame, or is it expanding?

If the Hydrogen was at rest initially  (which as you correctly point out for an expanding universe is actually a very reasonable assumption as the scale factor increases) then it just sits there and stays at rest for ever more.

Well, it's a hot universe at first under great pressure, meaning particles are not stationary, and they do experience force as they bump into each other. It should all cancel out, but not perfectly, and the anomalies, especially early on in such close quarters, introduce variations in local mass density, which, via gravity, tends to amplify the variations as gravity wells begin to form. In short, the homogeneous state is in sort of an unstable equilibrium.

However, applying the shell theorem (as it was done in the OP) can get a different answer.   The Hydrogen can be shown to have a net force in the direction of the origin.

That's a coordinate effect, as stated above. Coordinate acceleration is coordinate system dependent, and can be zero in some, such as the coordinate system putting the origin at the particle in question.  Yes, stuff will be attracted to each other. The expansion (if there is any) should slow measurably. If all is stationary, I think collapse occurs, even in Newtonian physics.

Keep in mind me not being an authority on this and I, like you, and just trying to world this out.

[Halc]

Dealing with 2nd spoiler:

Sure, but it is alto attracted in all other directions equally since there is a similar sphere at which it is on the edge, in any direction. The particle doesn't accelerate since the net force is zero.

More and more, I think my reply there doesn't hold water. I'm willing to say it is spherically symmetric. The inability to find a center of mass, or to identify a gravitational gradient (not a frame dependent thing) seems a better way to disqualify the applicability of the shell theorem in this Newtonian analysis. The particle doesn't accelerate since there's no downhill (a region of lower potential) available to it.
That said, the stuff does attract its neighbors and if initially all stationary, I think the density goes up over time. That very much does lower the gravitational potential, so everything (and not just some preferred thing) is going down hill.

You seem to agree that the shell theorem would show a non-zero net force exists in some direction.   You also seem to agree that direction is arbitrary.    The last sentence doesn't need to follow though.     You don't apply the Shell theorem with all possible origins and then sum all predicted net forces to obtain the actual real and very final net force.

Selecting an origin is a coordinate thing. You can't select more than one at a time, so adding up the forces from different frames doesn't work.

Worse that that, even it's magnitude is arbitrary  (just move the origin further away but in the same direction,   the inner sphere grows ~ r³ in volume and hence in mass,  while the gravitational force from it falls off only as ~1/r²,   making the total force vary ~ r  = the distance from our test particle to the origin).

Just so, which is why distant star are receding (and have greater acceleration) than the nearby ones.

You sometimes make arguments that the space isn't suitably spherically symmetric

I'm backing away from that.

All that exists in the space is the stuff we have stipulated will have a constant density, ρ, everywhere.   There are no points in space where a lack of spherical symmetry is exhibited.

Backing away because that's a pretty good counter to my comment.

I can similarly use the shell theorem to demonstrate that all clocks are infinitely dilated, and thus effectively stopped relative to 'the rate at which time actually goes', like that phrase means anything. I've brought that argument up elsewhere, and was punted out the door with a reply about Killing fields that I didn't understand. Takeaway is that I need to know GR better before asking questions like that. 

Example, trajectories of stars in a galaxy can be estimated by considering only that mass which is in a spherical region interior to that star etc.

This is incorrect since a galaxy isn't spherically symmetric. With a disk, a particle is attracted to rings of material outside its radius as well as the material inside. So it all needs to be considered. Kudos to the guys that work out where all the mass needs to be to get the velocity curves we see. It's a lot of calculus to do.

Anyway, just to be clear, what you're suggesting is the following:
A particle in the interior of a shell experiences no force due to gravity from the material of that shell.   That holds for shells of some finite thickness (assuming the usual things like spherical symmetry).   However, when that shell is so thick that it's of unbounded extent, the result may not hold, the particle might experience a force.

I cannot say I suggest this, at least not anymore. Like I said, there are reason the shell theorem doesn't apply, but this logic seems faulty.
Yes, our particle is attracted to an arbitrary 'origin' placed anywhere you like. That doesn't imply an objective force in a direction, only a coordinate frame dependent force. And as you've worked out, the greater the distance to the selected origin, the greater the coordinate acceleration towards it.

Hope this helps. Great topic, since it's rare to get one that gets me thinking.

[Halc]

Much the same question was asked [on stack exchange]

Cheat! The fun was in working it out ourselves.

Still, time to do that I guess. I have great respect for stack exchange in such matters.

n the negative side,  I don't especially agree with the highest ranked answer.   Their response pivots around the inability to identify a unique gravitational potential function, Φ, in such a space.

That sounds really strong to me. Gravitational potential is not frame dependent, and thus seems to be flat everywhere in this universe, hence no vector (direction) to any physical force.
The stack exchange question is even posed in the framework of Newtonian physics.

I'm taking more time to grok the rest about Gauss' law and all.

The other answers vary in quality and relevance.   There is one about some history and how major figures like Newton may also have noticed this problem that the shell theorem produces in this situation.

That would be quite relevant to this discussion, especially given the way the question was framed.

[alancalverd]

there is no reason why a well defined gravitational field g can't exist throughout space

There is no reason why fairies should not exist, but in the absence of the Mother of All Fairies, they can't. And in the absence of a density gradient, you can't have a gravitational field because there is nowhere for the vectors to go.

∇anything is meaningless, ergo no field, in an infinite homogeneous medium.

[Eternal Student (OP)]

Hi.

There is no reason why fairies should not exist,..... (etc.)....

    I'm fairly sure that Newton imagined gravity as a force that a unit mass experiences when it is put somewhere.    So to put that into modern terminology, he would assume the existance of a vector field, g(x) that exists throughout space.
    There can be vector fields for which no potential function exists.   We had a discussion (maybe 6 months+ ago) about some fluid flows in fluid dynamics.   There you can have a perfectly well defined vector field for the flow but it just turns out that the flow is rotational (has non-zero curl) which you could describe another way - you can say the field is non-conservative.    Specifically, there is no scalar field you could find such that the flow would be the gradient of that potential.
    I don't recall Newton ever saying that gravity was the gradient of a potential.   It is just the force that a unit mass would experience when it is put somewhere.   Don't get me wrong, in typical situations there will be a potential you can identify but we've got to try and go from the most elementary definitions of gravity.   It is a force on a unit mass not a gradient in some potential.

   Unlike fairies, we can write down a perfectly well defined vector field throughout space which has non-zero curl.  The vector field for what is called forced vortex flow is one such example.   That would give you a vector defined everywhere (and uniquely, there is one vector at any one point), which could describe a force that a unit mass experiences at that point.   However, since it has non-zero curl, there would be no potential function you could associate with it. 

 - - - - - - - - - - - - - - - - -

The good news    (It's fairly good news for me and at least non-harmfull to everyone else).
     I think I've think I've found something that I'm prepared to accept as being where the Shell theorem breaks down.
I still think the proof of the shell theorem outlined in post #11 is solid.   Starting from Gauss' law for gravity you get the shell theorem and you don't need to know anything about a scalar potential or most other things.   So the problem ( I think) is before that - just getting to Gauss' law.
    There still isn't any LaTeX so we're not having any mathematical notation.  So all I can do is point to some webpage.    Anyway, let us assume that Newton described gravity with    F =  GMm/r²    and   not that Gauss' law is the description for gravity.    So we need to get Gauss' law for gravity from Newton's law.
    This section in Wikipedia presents an outline proof for this:    https://en.wikipedia.org/wiki/Gauss%27s_law_for_gravity#Deriving_Gauss's_law_from_Newton's_law
      Note that near the top of their boxed section they express  g   as a volume integral.    The integrand has size that varies  ~ 1/r²    and that is not sufficient to ensure a volume integral over an inifinite volume would converge.   1/r³ would do it,  1/r² will not.    So that integral only converges when the density term ρ(r) can keep the integrand under control  (e.g.  if   ρ --> 0   like  1/r,  that would be just fine).
     In our situation, we have a constant density and hence the gravitational field g is undefined by this method.   The rest of the proof is then basically worthless,  you can't (for example) do the next line and find the divergence of an undefined thing.     Overall you just do not have the ability to assume Gauss' law for gravity would hold in this infinite homogeneous space.
    There are a few other methods for obtaining Gauss' law from Newton's law but the few I've glanced all run into problems.

    Anyway, that would mean that the paraphrased version of  @Halc 's comments that first appeared in  post#11 may be correct.

   Anyway, just to be clear, what you're suggesting is the following:
A particle in the interior of a shell experiences no force due to gravity from the material of that shell.   That holds for shells of some finite thickness (assuming the usual things like spherical symmetry).   However, when that shell is so thick that it's of unbounded extent, the result may not hold.

Best Wishes.

[Petrochemicals]

Has my post been deleted, and Paul's?

Being as you have reached a solution, if you could clarify a bit about she'll theory. If an object within a shell has no gravitational  effect on any object within it, surely that is from the point of view of the outside Observer? One would assume that the shell would indeed have gravitational attraction, if the she'll where 100, 000km thick and 10 million in diameter?