That is a good question, and one that I don’t have an answer to. Is there one? (you can’t?)
You look at the sanity of the theory. How do you tell a sane academic paper from one written by a crackpot (say, you don’t look at where it’s published and you don’t know the author)? Certainly you don’t need to go check experimental results or proofs in overwhelming majority of cases. Doing this reliably where absurdity heuristic breaks is claimed as a large part of the Art of rationality. It’s necessarily self-referential.
How do you tell a sane academic paper from one written by a crackpot (say, you don’t look at where it’s published and you don’t know the author)?
Simple—I don’t read it either way. There are plenty of papers written by folks I’ve heard of or in trusted publications to fill up all of the free time I don’t have.
if the ideas seem novel and useful, I expect empirical evidence. If there is none, I withhold judgement until there is.
Between crackpots and valid academic papers, out of context? Yes, I do deny that. Take Louis Savain (please!). I actually have a lot of the same intuitions about the nature of spacetime, and the problems involving even talking about time travel outside of a fictional context. Savain is clearly a crackpot—it’s practically painted all over his site. But if I had written a paper discussing some of these intuitions, and nobody noticed that it wasn’t published in a peer-reviewed journal nor that I’m not a physicist, then I don’t think there would be a clear way to notice that I was wrong.
Of course, a physicist could happily come along and say, “No, our theory is experimentally verified, and yours doesn’t even make sense. Here are references to the relevant evidence, and some engineering applications you might not have been aware of”.
ETA: I admit, I had “speak of the devil” in mind when I posted this. But it had to be done!
You are not doing the categorization as a civil service, you are doing it to efficiently build on gained understanding. If you are not ready to understand, you shouldn’t try to do that. But when you are, you play the role of that very physicist, you don’t need to see the crackpot’s experimental results.
you don’t need to see the crackpot’s experimental results.
That’s because the physicist has her own experimental results already, and the crackpot has none. If the crackpot has experimental results to back up his theory, then the physicist had bloody well better look at them! (if the paper is even being taken seriously enough to be read in the first place)
The one crackpot I interacted most strongly did have experimental results, and trumpeted them loudly. The experiment turned out to be a notoriously finnicky one (not quite down to Millikan experiment territory) done in slipshod fashion. This was utterly predictable, given purely theoretical considerations and examination of his style, even before it came to the observations—his theory contradicted, say, the existence of comets.
Experiments can be wrong. Maybe even most attempts at experiments are wrong. What makes a scientist a scientist instead of a crackpot is the debugging and validation. Trying to exclude every way the results might not mean what it seems like they mean—not just doing control-experiment comparison and saying you’ve done your duty.
Crackpot experiments, lacking these extra checks, are worthless.
Experiments can be wrong. Maybe even most attempts at experiments are wrong.
This wouldn’t surprise me much, at least in physics. There are probably more physics students than professional physicists, and those students do lots of tabletop experiments, badly. (My own old lab books document a refractive index measurement of −19.6, a disproof of the equivalence principle, and a laser beam that travelled at (1.05±0.01)c.) Nonetheless...
What makes a scientist a scientist instead of a crackpot is the debugging and validation. Trying to exclude every way the results might not mean what it seems like they mean—not just doing control-experiment comparison and saying you’ve done your duty.
...this is a bit too strong a distinction between crackpots & non-crackpots, though your basic point is right. The way I’d put it: a non-crackpot confronted with a bizarre result immediately wonders, “what did I do wrong?”, but a crackpot confronted with the same result immediately gasps, “I knew it!”.
I guess I’m just paraphrasing Dear Leader, really: one’s strength as a non-crackpot is one’s ability to be more confused by bizarre, inexplicable results than predictable results.
What makes a scientist a scientist instead of a crackpot is the debugging and validation. Trying to exclude every way the results might not mean what it seems like they mean—not just doing control-experiment comparison and saying you’ve done your duty.
On a completely unrelated note, screw the Millikan experiment. That one lab where we had to replicate it in undergrad with the world’s shittiest equipment is probably the only reason I’m a mathematician and not a physicist.
My high school physics class took the Millikan experiment to a new level: we installed a calculator program in which oil drops were simulated by pixels moving down the screen, and you could press buttons to vary the simulated electric charge.
I wonder if I can blame becoming a mathematician on that, too.
[Someone wrote to tell me that my name was used on this forum]
Let’s see now. I use irrefutable logic to show that Stephen Hawking is full of shit when he claims that time travel is a valid scientific pursuit and that it is not forbidden by General Relativity. I take the position that Sir Karl Popper was correct when he compared Einstein’s spacetime to “Parmenides’ block universe in which nothing happens.” If nothing can move in spacetime, how can anybody claim that GR does not forbid time travel?
Now who is the crackpot, I or Hawking?
Blake, you don’t have a monopoly on ad hominems, you know. I, too, am free to insult people like you as I see fit. And I admit that I rather enjoy it. You are a gutless, politically correct sycophant, in my opinion. Like most of the commenters on this forum, you are here to kiss Yudkowsky’s ass. Eliezer, on his part, writes mostly to kiss the asses of those whom he considers his superiors in the scientific community. You’re all a bunch of ass kissers. How about that?
You all talk about rationality as if somehow you were born with an exclusive monopoly on logic and wisdom but it remains that you are all members of the human race, with all the biases and dishonesty and cowardice that it entails. It takes guts and gigantic huevos to be rational in this world. Ass kissers don’t have those things. I’d rather be a crackpot in your eyes than an ass kisser. See you around.
You look at the sanity of the theory. How do you tell a sane academic paper from one written by a crackpot (say, you don’t look at where it’s published and you don’t know the author)? Certainly you don’t need to go check experimental results or proofs in overwhelming majority of cases. Doing this reliably where absurdity heuristic breaks is claimed as a large part of the Art of rationality. It’s necessarily self-referential.
Simple—I don’t read it either way. There are plenty of papers written by folks I’ve heard of or in trusted publications to fill up all of the free time I don’t have.
if the ideas seem novel and useful, I expect empirical evidence. If there is none, I withhold judgement until there is.
This doesn’t sound realistic. Do you deny your ability to tell the difference?
Between crackpots and valid academic papers, out of context? Yes, I do deny that. Take Louis Savain (please!). I actually have a lot of the same intuitions about the nature of spacetime, and the problems involving even talking about time travel outside of a fictional context. Savain is clearly a crackpot—it’s practically painted all over his site. But if I had written a paper discussing some of these intuitions, and nobody noticed that it wasn’t published in a peer-reviewed journal nor that I’m not a physicist, then I don’t think there would be a clear way to notice that I was wrong.
Of course, a physicist could happily come along and say, “No, our theory is experimentally verified, and yours doesn’t even make sense. Here are references to the relevant evidence, and some engineering applications you might not have been aware of”.
ETA: I admit, I had “speak of the devil” in mind when I posted this. But it had to be done!
You are not doing the categorization as a civil service, you are doing it to efficiently build on gained understanding. If you are not ready to understand, you shouldn’t try to do that. But when you are, you play the role of that very physicist, you don’t need to see the crackpot’s experimental results.
That’s because the physicist has her own experimental results already, and the crackpot has none. If the crackpot has experimental results to back up his theory, then the physicist had bloody well better look at them! (if the paper is even being taken seriously enough to be read in the first place)
The one crackpot I interacted most strongly did have experimental results, and trumpeted them loudly. The experiment turned out to be a notoriously finnicky one (not quite down to Millikan experiment territory) done in slipshod fashion. This was utterly predictable, given purely theoretical considerations and examination of his style, even before it came to the observations—his theory contradicted, say, the existence of comets.
Experiments can be wrong. Maybe even most attempts at experiments are wrong. What makes a scientist a scientist instead of a crackpot is the debugging and validation. Trying to exclude every way the results might not mean what it seems like they mean—not just doing control-experiment comparison and saying you’ve done your duty.
Crackpot experiments, lacking these extra checks, are worthless.
This wouldn’t surprise me much, at least in physics. There are probably more physics students than professional physicists, and those students do lots of tabletop experiments, badly. (My own old lab books document a refractive index measurement of −19.6, a disproof of the equivalence principle, and a laser beam that travelled at (1.05±0.01)c.) Nonetheless...
...this is a bit too strong a distinction between crackpots & non-crackpots, though your basic point is right. The way I’d put it: a non-crackpot confronted with a bizarre result immediately wonders, “what did I do wrong?”, but a crackpot confronted with the same result immediately gasps, “I knew it!”.
I guess I’m just paraphrasing Dear Leader, really: one’s strength as a non-crackpot is one’s ability to be more confused by bizarre, inexplicable results than predictable results.
This would make most modern professional scientists crackpots which sounds a bit noncentral—they may be no true scientists, but they seem very different from the crackpots I’ve met.
There’s a bit of a gap between what ordinary not-very-good scientists do to make sure the experiment is right and what they should be doing.
There is a colossal gulf between what crackpots do and ordinary not-very-good scientists do.
Certain branches of physics are nowhere near as bad as medicine when it comes to that.
On a completely unrelated note, screw the Millikan experiment. That one lab where we had to replicate it in undergrad with the world’s shittiest equipment is probably the only reason I’m a mathematician and not a physicist.
My high school physics class took the Millikan experiment to a new level: we installed a calculator program in which oil drops were simulated by pixels moving down the screen, and you could press buttons to vary the simulated electric charge.
I wonder if I can blame becoming a mathematician on that, too.
/me stares in horror.
Undergrad. Millikan. Oil. Drop. Experiment.
shiver
[Someone wrote to tell me that my name was used on this forum]
Let’s see now. I use irrefutable logic to show that Stephen Hawking is full of shit when he claims that time travel is a valid scientific pursuit and that it is not forbidden by General Relativity. I take the position that Sir Karl Popper was correct when he compared Einstein’s spacetime to “Parmenides’ block universe in which nothing happens.” If nothing can move in spacetime, how can anybody claim that GR does not forbid time travel?
Now who is the crackpot, I or Hawking?
Blake, you don’t have a monopoly on ad hominems, you know. I, too, am free to insult people like you as I see fit. And I admit that I rather enjoy it. You are a gutless, politically correct sycophant, in my opinion. Like most of the commenters on this forum, you are here to kiss Yudkowsky’s ass. Eliezer, on his part, writes mostly to kiss the asses of those whom he considers his superiors in the scientific community. You’re all a bunch of ass kissers. How about that?
You all talk about rationality as if somehow you were born with an exclusive monopoly on logic and wisdom but it remains that you are all members of the human race, with all the biases and dishonesty and cowardice that it entails. It takes guts and gigantic huevos to be rational in this world. Ass kissers don’t have those things. I’d rather be a crackpot in your eyes than an ass kisser. See you around.