The same holds true for black holes. If you see a star that is rapidly shifting red and blue (moving away then moving closer) you don't need to see the black hole to know it's there.

But you do not know that there is a Black Hole there. You only know that there is something very massive there.

The idea that a Black Hole can exist and contain a mathematical singularity that can somehow conserve massiveness is not natural. It is not possible in nature. It is only in our imagination that such things can exist. So, if we want to see how this relates to the real world we must conclude that there is some unknown amount of massive density at the centre of most, and maybe all, galaxies that pulls in matter and spews out radiation and ionic debris.

Edit: I am led to this speculation because I can not model the formation of a Black Hole by incrementally adding mass to a local area. If someone can do that I would like to see the equations.