Ok, that’s a lot of questions. I’ll do my best, but I have to tell you that your quest is, in my opinion, a bit quixotic.
What is the reason for the existence of the theory of the charm quark (or any generation 2-3 quark)? What are some results of experiments that necessitate the existence of a charm quark?
Basically the strange quark is motivated by the existence of kaons, charm quarks by the D family of mesons (well, historically the J/psi, but I’m more familiar with the D mesons), and beauty quarks by the B family. As for truth quarks, mainly considerations of symmetry. Let’s take kaons, the argument being the same for the other families. If the kaon were to decay by the strong force, it would be extremely short-lived, because it could go pretty immediately to two pions; there would certainly be no question of seeing it in a tracking detector, the typical timescale of strong decays being 10^-23 seconds. Even at lightspeed you don’t get far in that time! We therefore conclude that there is some conservation principle preventing the strong decay, and that the force by which the kaon decays does not respect this conservation principle. Hence we postulate a strange quark, whose flavour (strangeness) is conserved by the strong force (so, no strange-to-up (or down) transition at strong-force speeds) but not by the weak force.
I should note that quark theory has successfully predicted the existence of particles before they were observed; you might Google “Eightfold Path” if you’re not familiar with this history, or have a look at the PDG’s review. (Actually, on closer inspection I see that the review is intending for working physicists familiar with the history—it’s not an introduction to the Eightfold Path, per se. Probably Google would serve you better.)
Which of the known hadrons can be directly observed in any way, as opposed to theorized as a mathematical in-between or as a trigger for some directly observable decay?
For this I have to digress into cross-sections. Suppose you are colliding an electron and a positron beam, and you set up a detector at some particular angle to the beam—for example, you can imagine the detector looking straight down at the collision point:
___detector
e+ -----> collision <------- e-
Now, the cross-section (which obviously is a function of the angle) can be thought of as the probability that you’ll see something in the detector. If electron and positron just glance off each other without annihilating (at relativistic speeds this can easily happen—they have to get pretty close to interact, and our control of the beams is only so good), we call that Bhabha scattering, and it has a particular cross-section structure. For obvious reasons, the cross-section is highest at small angles; that is, it is really quite unlikely for the electron and positron to dance past each other in the exact way that throws them out at a ninety-degree angle to their previous paths; but it’s pretty easy for them to give each other a one-degree kick. If you calculate the cross-section at some particular angle as a function of the total beam energy, you’ll see that the higher the energy, the lower the cross-section, and indeed experiment confirms this.
What if the electron and positron do annihilate, creating a virtual photon that then decays to some other pair of particles—for example, a charm-anticharm pair? Well, again, the cross-section is highest near the beam (basically to conserve the angular momentum—you have to do spin math) and decreases with energy.
So we have this cross-section that decreases monotonically with energy. However, as you run your beam energy up, at very specific energies you will see a sharp increase and drop-off, in a classic Breit-Wigner shape. In other words, at some particular energy it suddenly becomes much more likely that your decay products get kicked away from the beam. Why is that? We refer to these bumps in the spectrum as resonances, and explain them by appealing to bound states—particles, in other words. What happens is that with an intermediate bound state, there are additional Feynman paths that open up between the initial state “electron and positron” and the final state “hit in detector at angle X”. Additional paths through parameter space gives you additional probability unless you’re very unlucky with the phases, hence the bump in the cross-section—the final state becomes more likely. (Additionally, for reasons of spin math that I won’t go into here, the decay products from a bound state of two quarks are produced much more isotropically than back-to-back quark-antiquark pairs.)
Here’s a different way of looking at it. Suppose you have a detector that encloses the collision space, so you can reconstruct most of the decay products; and you decide to take all pion pairs and calculate “If these two particles came from a common decay, what was the mass of the particle that decayed?” Then this spectrum will basically be flat, but you will get an occasional peak at specific masses. Again, we explain this by appeal to a bound state.
It occurs to me that this may not actually differ from what you call “mathematical in-betweens”; I have answered as though this phrase refers to virtual particles, which are indeed a bit of a convenient fiction. Anyway, this is why we believe in the various hadrons and mesons.
(I’m getting “comment too long” errors; splitting my answer here.)
Ok, that’s a lot of questions. I’ll do my best, but I have to tell you that your quest is, in my opinion, a bit quixotic.
Basically the strange quark is motivated by the existence of kaons, charm quarks by the D family of mesons (well, historically the J/psi, but I’m more familiar with the D mesons), and beauty quarks by the B family. As for truth quarks, mainly considerations of symmetry. Let’s take kaons, the argument being the same for the other families. If the kaon were to decay by the strong force, it would be extremely short-lived, because it could go pretty immediately to two pions; there would certainly be no question of seeing it in a tracking detector, the typical timescale of strong decays being 10^-23 seconds. Even at lightspeed you don’t get far in that time! We therefore conclude that there is some conservation principle preventing the strong decay, and that the force by which the kaon decays does not respect this conservation principle. Hence we postulate a strange quark, whose flavour (strangeness) is conserved by the strong force (so, no strange-to-up (or down) transition at strong-force speeds) but not by the weak force.
I should note that quark theory has successfully predicted the existence of particles before they were observed; you might Google “Eightfold Path” if you’re not familiar with this history, or have a look at the PDG’s review. (Actually, on closer inspection I see that the review is intending for working physicists familiar with the history—it’s not an introduction to the Eightfold Path, per se. Probably Google would serve you better.)
For this I have to digress into cross-sections. Suppose you are colliding an electron and a positron beam, and you set up a detector at some particular angle to the beam—for example, you can imagine the detector looking straight down at the collision point:
___detector
e+ -----> collision <------- e-
Now, the cross-section (which obviously is a function of the angle) can be thought of as the probability that you’ll see something in the detector. If electron and positron just glance off each other without annihilating (at relativistic speeds this can easily happen—they have to get pretty close to interact, and our control of the beams is only so good), we call that Bhabha scattering, and it has a particular cross-section structure. For obvious reasons, the cross-section is highest at small angles; that is, it is really quite unlikely for the electron and positron to dance past each other in the exact way that throws them out at a ninety-degree angle to their previous paths; but it’s pretty easy for them to give each other a one-degree kick. If you calculate the cross-section at some particular angle as a function of the total beam energy, you’ll see that the higher the energy, the lower the cross-section, and indeed experiment confirms this.
What if the electron and positron do annihilate, creating a virtual photon that then decays to some other pair of particles—for example, a charm-anticharm pair? Well, again, the cross-section is highest near the beam (basically to conserve the angular momentum—you have to do spin math) and decreases with energy.
So we have this cross-section that decreases monotonically with energy. However, as you run your beam energy up, at very specific energies you will see a sharp increase and drop-off, in a classic Breit-Wigner shape. In other words, at some particular energy it suddenly becomes much more likely that your decay products get kicked away from the beam. Why is that? We refer to these bumps in the spectrum as resonances, and explain them by appealing to bound states—particles, in other words. What happens is that with an intermediate bound state, there are additional Feynman paths that open up between the initial state “electron and positron” and the final state “hit in detector at angle X”. Additional paths through parameter space gives you additional probability unless you’re very unlucky with the phases, hence the bump in the cross-section—the final state becomes more likely. (Additionally, for reasons of spin math that I won’t go into here, the decay products from a bound state of two quarks are produced much more isotropically than back-to-back quark-antiquark pairs.)
Here’s a different way of looking at it. Suppose you have a detector that encloses the collision space, so you can reconstruct most of the decay products; and you decide to take all pion pairs and calculate “If these two particles came from a common decay, what was the mass of the particle that decayed?” Then this spectrum will basically be flat, but you will get an occasional peak at specific masses. Again, we explain this by appeal to a bound state.
It occurs to me that this may not actually differ from what you call “mathematical in-betweens”; I have answered as though this phrase refers to virtual particles, which are indeed a bit of a convenient fiction. Anyway, this is why we believe in the various hadrons and mesons.
(I’m getting “comment too long” errors; splitting my answer here.)