It is not a trivial question and depends upon the actual geometry of the situation but take the simplest model.

Imagine the pall bearers (A,B,C,D, clockwise) are of equal heights and the 'load' consists of a mass where the diagonals cross (i.e. the middle of the box) and it is on the floor of the box. They will be carrying 1/4 of the load each and you can replace them with 4 spring balances. Raising one corner (A) will increase the load on that corner because, if you take moments about the opposite corner (C), you will be reducing the loads on the other two (B,D) corners. If you take moments about a line through the corners (B,D) the load on the corner (C) will also increase, to be the same as the load at (A). The total of the four loads will always be the same.

In the end, if you lift high enough, you and your diagonal partner (A and C) would be lifting the whole load between you and the other diagonal loads (B and D)would be zero (apart from a bit of force to balance the arrangement a bit.

Real people are not the same as spring balances and will react to changes by compensating, raising their arms and even pushing and pulling laterally to avoid the load sliding. Also, the centre of mass of the load would be above the plane of the shoulders of the bearers, so raising one end or one side will tip the CM in the direction of the lower bearer(s) and increase their load. The effect 'side to side' would be greater than the effect ' end to end' because the angle of tilt would be greater for a given difference in bearer height.

Having the two short ones one end and the two tall ones the other end would be 'fairest'.