You mis-understand me. The air in the balloon is compressed because the surrounding air pressure increases as you go lower ( the air around it is compressed, too of course, by the atmosphere above it). The hole in the bottom and the flexible envelope allows air in an out and equalise the pressure. It is the relative densities of the warm air inside and the cool air outside that gives you the lift.
A hot air balloon is, theoretically, quite a stable object, as far as its height is concerned.
As your height decreases, the gravitational PE of the ballon + its air decreases and the temperature of the air inside MUST rise - to obey the energy conservation law, the warmer air will give you more lift and slow you down. If you let just a little hot air out, the balloon will drop a short distance until it reaches a new equilibrium height. It would be possible to let just enough hot air out and to land at zero speed - but difficult to judge.
The reason for all the frantic burning of gas is that you need to do things quickly - to get down where you want on a steep path (avoiding trees and power lines). You have to build up speed and then apply the 'brakes' to avoid ploughing into the ground.
You don't need an envelope to see this effect; a (cumulus) cloud is formed when a body of air warms up near the ground. It expands and rises until its temperature drops (due to the gas laws) to bring its density equal to the surrounding air - it reaches a stable ceiling where the thermal energy it got at the ground is all transferred into gravitational potential energy. The hotter the ground, the higher it reaches.