I've used both extensively. Mathematica is better for theory work, since it handles analytic formulae much better--it has powerful tools for carrying through computations entirely symbolically. It is, however, slow and cumbersome to use for numerical simulations and works poorly as a programming language. Matlab is just the opposite: it is cumbersome to use it for extensive analytic work, but is very similar to most programming languages in syntax (if you know how to code, Matlab takes some learning, but the basics are the same). Matlab is also extremely good at discrete math simulations (everything is matrices or arrays of numbers).