They don't annihilate. Nor do electrons and protons. The latter are attracted towards one another just like electrons and positrons. Electrons and positrons annihilate because they're both simple spinors but with the opposite chirality or "handedness". See this GNUFDL image by Slawekb on the Wikipedia spinor article:


A spinor visualized as a vector pointing along the Möbius band, exhibiting a sign inversion when the circle (the "physical system") is rotated through a full turn of 360°

Make two Mobius strips. For the first one twist the paper clockwise before taping the ends. For the second one twist the paper anticlockwise before taping the ends. You then have two Mobius strips with the opposite chirality. The electron is akin to one, the positron is akin to another. They annihilate because they're the same apart from their opposite chirality. However a proton is very different to an electron. It doesn't just have the opposite chirality. It won't annihilate with an electron, just as your Chubb key won't fit your Yale lock. Ditto for a positron. What you need to annihilate the proton, is an antiproton:


Image credit CSIRO

IMHO you take a look at topological quantum field theory to pursue this. It's interesting stuff.