But, if we are always adding leap seconds.

Then that means that our atomic clocks are running slower than the spinning of the Earth, whether or not the Earth changes slightly on a day-to-day basis. On a long term basis, one can synchronize the seconds to represent the rotation of the Earth on a decade or century basis.

I misspoke above. If the clocks are running slow (on average), then the Earth is running fast (on average). That can not be accounted for by the predicted Earth's long-term rate of rotation slowing down over time. However, perhaps in a million years, the clocks will be more predictive of the actual diurnal cycle.

However, redefining the second would also mean redefining the speed of light, and the meter. Not a big deal for most of us though, as our tape measures won't change.