Lorentz force is explained or demonstrated using Fleming's Left Hand Rule or the Force equation using cross product.

What I'm trying to ask here is, why is the direction fixed using the Left Hand Rule or the equation? Why is the direction here and not there?

Equation and rules are the easiest way to explain the force direction, but they don't explain why.

Explaining why is always a much more costly and challenging task. It is usually, also, open for different interpretations, which for me, gives the quantum theory most of its beauty. Mind you, those interpretations cannot be at the cost of an incorrect blueprint of experimental confirmation.

Remember, we use these rules as an attempt to fit our equations into reality, we don't make reality fit into the equations.

You will find that the right hand rule is in fact one of two solutions in fact. There is also a left hand rule. It just so happens that rotations lead to these two results, and mathematically constructed to make sense of these two different rules. Since mathematics can almost be applied to anything, it is by no mere coincidence we have decided it to be a most proficient tool.