Forget the word “annihilation”.
Particles interact. The interaction often results in new particles.
For instance, a neutron may interact with an electron-neutrino. The result of this interaction is a proton and an electron. Would you call this “annihilation”? I guess not. But fundamentally this is no different in principle from the interaction of an electron and a positron, which is supposedly the classic “annihilation” scenario, producing a pair of photons.
So then, why do, say, an electron and a proton not “annihilate”? Well… as a matter of fact, they might. But this is not a reaction that happens often. Why? The answer is energy. You see, if a proton and an electron “annihilate”, the resulting particles are a neutron and an antielectron-neutrino. Now here is the problem: The mass-energy of a neutron is more than the mass-energy of a proton and an electron. Hence, this interaction can only take place if the proton and the electron have sufficient kinetic energy to make up for the deficit.
This is, in fact, why two photons do not “annihilate”, despite being antiparticles of each other. The product of “annihilation” could be, for instance, an electron-positron pair, but the electron-positron pair has substantial mass-energy. So the photons must have enough kinetic energy, which means hard X-ray/gamma-ray photons. Anything less (e.g., visible light) and there just isn’t enough mass-energy. Not even close.
Electrons and positrons annihilate easily because the resulting photons have no rest mass and can have arbitrarily little energy. But, to give another example, neutrinos and antineutrinos cannot annihilate this easily. They do not interact electromagnetically, so they cannot produce photons. The mediating particles of the weak interaction, the W- and Z-bosons, are incredibly heavy. So the neutrinos would have to have immense energies in order to be able to produce these as annihilation products.