supported by CEEALAR

I happened to glance over at the CEEALAR bookshelf one day and saw a book called Craft of Research. Craft of Research is a book by 5 english professors that tries to provide a field-agnostic guide to, well, doing research. I previously shared my notes piecemeal in my lesswrong shortform, and today I’m collecting them into one top-level post. There will occasionally be reflections on the relevance of this book to the alignment community throughout.

The audience models of research—thoughts on chapter 2

Writers can’t avoid creating some role for themselves and their readers, planned or not

Before considering the role you’re creating for your reader, consider the role you’re creating for yourself. Your broad options are the following

I’ve found some new and interesting information—I have information for you
I’ve found a solution to an important practical problem—I can help you fix a problem
I’ve found an answer to an important question—I can help you understand something better

The authors recommend assuming one of these three. There is of course a wider gap between information and the neighborhood of problems and questions than there is between problems and questions! Later on in chapter four the authors provide a graph illustrating problems and questions: Practical problem -> motivates -> Research question -> defines -> Conceptual/research problem. Information, when provided mostly for novelty, however, is not in this cycle. Information can be leveled at problems or questions, plays a role in providing solutions or answers, but can also be for “its own sake”.

I’m reminded of a paper/post I started but never finished, on providing a poset-like structure to capabilities. I thought it would be useful if you could give a precise ordering on a set of agents, to assign supervising/overseeing responsibilities. Looking back, providing this poset would just be a cool piece of information, effectively: I wasn’t motivated by a question or problem so much as “look at what we can do”. Yes, I can post-hoc think of a question or a problem that the research would address, but that was not my prevailing seed of a reason for starting the project. Is the role of the researcher primarily a writing thing, though, applying mostly to the final draft? Perhaps it’s appropriate for early stages of the research to involve multi-role drifting, even if it’s better for the reader experience if you settle on one role in the end.

Additionally, it occurs to me that maybe “I have information for you” mode just a cheaper version of the question/problem modes. Sometimes I think of something that might lead to cool new information (either a theory or an experiment), and I’m engaged moreso by the potential for novelty than I am by the potential for applications.

I think I’d like to become more problem-driven. To derive possibilities for research from problems, and make sure I’m not just seeking novelty. At the end of the day, I don’t think these roles are “equal” I think the problem-driven role is the best one, the one we should aspire to.

[When you adopt one of these three roles, you must] cast your readers in a complementary role by offering them a social contract: _I’ll play my part if you play yours … if you cast them in a role they won’t accept, you’re likely to lose them entirely… You must report your research in a way that motivates your readers to play the role you have imagined for them.

The three reader roles complementing the three writer roles are

Entertain me
Help me solve my practical problem
Help me understand something better

It’s basically stated that your choice of writer role implies a particular reader role, 1 mapping to 1, 2 mapping to 2, and 3 mapping to 3.

Role 1 speaks to an important difficulty in the x-risk, EA, alignment community; which is how not to get drawn into the phenomenal sensation of insight when something isn’t going to help you on a problem. At my local EA meetup I sometimes worry that the impact of our speaker events is low, because the audience may not meaningfully update even though they’re intellectually engaged. Put another way, intellectual engagement can be goodhartable, the sensation of insight can distract you from your resolve to shatter your bottlenecks and save the world if it becomes an end itself. Should researchers who want to be careful about this avoid the first role entirely? Should the alignment literature look upon the first reader role as a failure mode? We talk about a lot of cool stuff, it can be easy to be drawn in by the cool factor like some of the non-EA rationalists I’ve met at meetups.

I’m not saying reader role number two absolutely must dominate, because it can diverge from deconfusion which is better captured by reader role number three.

Division of labor between reader and writer, writer roles do not always imply exactly one reader role

Isn’t it the case that deconfusion/writer role three research can be disseminated to practical (as opposed to theoretical) -minded people, and then those people turn question-answer into problem-solution? You can write in the question-answer regime, but there may be that (rare) reader who interprets it in the problem-solution regime! This seems to be an extremely good thing that we should find a way to encourage. In general reading the drifts across multiple roles seems like the most engaged kind of reading.

Questions and Problems—thoughts on chapter 4

Last time we discussed the difference between information and a question or a problem, and I suggested that the novelty-satisfied mode of information presentation isn’t as good as addressing actual questions or problems. In chapter 3 which I have not typed up thoughts about, A three step procedure is introduced

Topic: “I am studying …”
Question: ”… because I want to find out what/why/how …”
Significance: ”… to help my reader understand …” As we elaborate on the different kinds of problems, we will vary this framework and launch exercises from it.

Some questions raise problems, others do not. A question raises a problem if not answering it keeps us from knowing something more important than its answer.

The basic feedback loop introduced in this chapter relates practical with conceptual problems and relates research questions with research answers.

Practical problem -> motivates -> research question -> defines -> conceptual/research problem -> leads to -> research answer -> helps to solve -> practical problem (loop)

What should we do vs. what do we know—practical vs conceptual problems

Opposite eachother in the loop are practical problems and conceptual problems. Practical problems are simply those which imply uncertainty over decisions or actions, while conceptual problems are those which only imply uncertainty over understanding. Concretely, your bike chain breaking is a practical problem because you don’t know where to get it fixed, implying that the research task of finding bike shops will reduce your uncertainty about how to fix the bike chain.

Conditions and consequences

The structure of a problem is that it has a condition (or situation) and the (undesirable) consequences of that condition. The consequences-costs model of problems holds both for practical problems and conceptual problems, but comes in slightly different flavors. In the practical problem case, the condition and costs are immediate and observed. However, a chain of “so what?” must be walked.

Readers judge the significance of your problem not by the cost you pay but by the cost they pay if you don’t solve it… To make your problem their problem, you must frame it from their point of view, so that they see its cost to them.

One person’s cost may be another person’s condition, so when stating the cost you ought to imagine a socratic “so what?” voice, forcing you to articulate more immediate costs until the socratic voice has to really reach in order to say that it’s not a real cost.

The conceptual problem case is where intangibles play in. The condition in that case is always the simple lack of knowledge or understanding of something. The cost in that case is simple ignorance.

Modus tollens

A helpful exercise is if you find yourself saying “we want to understand x so that we can y”, try flipping to “we can’t y if we don’t understand x”. This sort of shifts the burden on the reader to provide ways in which we can y without understanding x. You can do this iteratively: come up with _z_s which you can’t do without y, and so on.

Pure vs. applied research

Research is pure when the significance stage of the topic-question-significance frame refers only to knowing, not to doing. Research is applied when the significance step refers to doing. Notice that the question step, even in applied research, refers to knowing or understanding.

Connecting research to practical consequences

You might find that the significance stage is stretching a bit to relate the conceptual understanding gained from the question stage. Sometimes you can modify and add a fourth step to the topic-question-significance frame and make it into topic-conceptual question-conceptual significance-possible practical application. Splitting significance into two helps you draw reasonable, plausible applications. A claimed application is a stretch when it is not plausible. Note: the authors suggest that there is a class of conceptual papers in which you want to save practical implications entirely for the conclusion, that for a certain kind of paper practical applications do not belong in the introduction.

AI safety

One characterisitic of AI safety that makes it difficult both to do and interface with is the chains of “so what” are often very long. The path from deconfusion research to everyone dying or not dying feels like a stretch if not done carefully, and has a lot of steps when done carefully. As I mentioned in the section on chapter 2, it’s easy to get sucked into the “novel information for it’s own sake” regime at least as a reader. More practical oriented approaches are perhaps those that seek new regimes for how to even train models, and the “so what?” is answered “so we have dramatically less OODR-failures” or something. The condition-costs framework seems really beneficial for articulating alignment agendas and directions.

Misc

“Researchers often begin a project without a clear idea of what the problem even is.”
Look for problems as you read. When you see contradictions, inconsistencies, incomplete explanations tentatively assume that readers would or should feel the same.
Ask not “Can I solve it?” but “will my readers think it ought to be solved?”
“Try to formulate a question you think is worth answering, so that down the road, you’ll know how to find a problem others think is worth solving.”

Sources—notes on chapters 5 and 6

Primary, secondary, and tertiary sources

Primary sources provide you with the “raw data” or evidence you will use to develop, test, and ultimately justify your hypothesis or claim. Secondary sources are books, articles, or reports that are based on primary sources and are intended for scholarly or professional audiences. Tertiary sources are books and articles that synthesize and report on secondary sources for general readers, such as textbooks, articles in encyclopedias, and articles in mass-circulation publications.

The distinction between primary and secondary sources comes from 19th century historians, and the idea of tertiary sources came later. The boundaries can be fuzzy, and are certainly dependent on the task at hand.

I want to reason about what these distinctions look like in the alignment community, and whether or not they’re important.

The rest of chapter five is about how to use libraries and information technologies, and evaluating sources for relevance and reliability.

Chapter 6 starts off with the kind of thing you should be looking for while you read

Look for creative agreement

Offer additional support. You can offer new evidence to support a source’s claim.
Confirm unsupported claims. You can prove something that a source only assumes or speculates about.
Apply a claim more widely. You can extend a position.

Look for creative disagreement

Contradictions of kind. A source says something is one kind of thing, but it’s another.
Part-whole contradictions. You can show that a source mistakes how the parts of something are related.
Developmental or historical contradictions. You can show that a source mistakes the origin or development of a topic.
External cause-effect contradictions. You can show that a source mistakes a causal relationship.
Contradictions of perspective. Most contradictions don’t change a conceptual framework, but when you contradict a “standard” view of things, you urge others to think in a new way.

The rest of chapter 6 is a few more notes about what you’re looking for while reading (evidence, reasons), how to take notes, and how to stay organized while doing this.

The alignment community

I think I see the creative agreement modes and the creative disagreement modes floating around in posts. Would it be more helpful if writers decided on one or two of these modes before sitting down to write?

Moreover, what is a primary source in the alignment community? Surely if one is writing about inner alignment, a primary source is the Risks from Learned Optimization paper. But what are Risks’ primary, secondary, tertiary sources? Does it matter?

Now look at Arbital. Arbital started off to be a tertiary source, but articles that seemed more like primary sources started appearing there. I remember distinctively thinking “what’s up with that?” it struck me as awkward for Arbital to change it’s identity like that, but I end up thinking about and citing the articles that seem more like primary sources.

There’s also the problem of stuff in the memeplex not written down is the real “primary” source while the first person who happens to write it down looks like they’re writing a primary source when in fact what they’re doing is really more like writing a secondary or even tertiary source.

Good arguments—notes on chapter 7

Arguments take place in 5 parts.

Claim: What do you want me to believe?
Reasons: Why should I agree?
Evidence: How do you know? Can you back it up?
Acknowledgment and Response: But what about … ?
Warrant: How does that follow?

This can be modeled as a conversation with readers, where the reader prompts the writer to taking the next step on the list.

Claims ought to be supported with reasons. Reasons ought to be based on evidence. Arguments are recursive: a part of an argument is an acknowledgment of an anticipated response, and another argument addresses that response. Finally, when the distance between a claim and a reason grows large, we draw connections with something called warrants.

The logic of warrants proceeds in generalities and instances. A general circumstance predictably leads to a general consequence, and if you have an instance of the circumstance you can infer an instance of the consequence.

Arguing in real life papers is complexified from the 5 steps, because

Claims should be supported by two or more reasons
A writer can anticipate and address numerous responses. As I mentioned, arguments are recursive, especially in the anticipated response stage, but also each reason and warrant can necessitate a subargument.

You might embrace a claim too early, perhaps even before you have done much research, because you “know” you can prove it. But falling back on that kind of certainty will just keep you from doing your best thinking.

Claims—thoughts on chapter 8

Broadly, the two kinds of claims are conceptual and practical.

Conceptual claims ask readers not to ask, but to understand. The flavors of conceptual claim are as follows:

Claims of fact or existence
Claims of definition and classification
Claims of cause and consequence
Claims of evaluation or appraisal

There’s essentially one flavor of practical claim

Claims of action or policy.

If you read between the lines, you might notice that a kind of claim of fact or cause/consequence is that a policy works or doesn’t work to bring about some end. In this case, we see that practical claims deal in ought or should. There is a difference, perhaps subtle perhaps not, between “X brings about Y” and “to get Y we ought to X”.

Readers expect a claim to be specific and significant. You can evaluate your claim along these two axes.

To make a claim specific, you can use precise language and explicit logic. Usually, precision comes at the cost of a higher word count. To gain explicitness, use words like “although” and “because”. Note some fields might differ in norms.

You can think of significance of a claim as the quantity it asks readers to change their mind, or I suppose even behavior.

While we can’t quantify significance, we can roughly estimate it: if readers accept a claim, how many other beliefs must they change?

Avoid arrogance.

As paradoxical as it seems, you make your argument stronger and more credible by modestly acknowledging its limits.

Two ways of avoiding arrogance are acknowledging limiting conditions and using hedges to limit certainty.

Don’t run aground: there are innumerable caveats that you could think of, so it’s important to limit yourself only to the most relevant ones or the ones that readers would most plausibly think of. Limiting certainty with hedging is given by example of Watson and Crick, publishing what would become a high-impact result, “We wish to suggest … in our opinion … we believe … Some … appear”

without the hedges, Crick and Watson would be more concise but more aggressive.

In most fields, readers distrust flatfooted certainty

It is not obvious how to walk the line between hedging too little and hedging too much.

thoughts on chapter 9

Getting the easy things right shows respect for your readers and is the best training for dealing with the hard things.

If they don’t believe the evidence, they’ll reject the reasons and, with them, your claim.

We saw previously that claims ought to be supported with reasons, and reasons ought to be based on evidence. Now we will look closer at reasons and evidence.

Reasons must be in a clear, logical order. Atomically, readers need to buy each of your reasons, but compositionally they need to buy your logic. Storyboarding is a useful technique for arranging reasons into a logical order: physical arrangements of index cards, or some DAG-like syntax. Here, you can list evidence you have for each reason or, if you’re speculating, list the kind of evidence you would need.

When storyboarding, you want to read out the top level reasons as a composite entity without looking at the details (evidence), because you want to make sure the high-level logic makes sense.

Readers will not accept a reason until they see it anchored in what they consider to be a bedrock of established fact. … To count as evidence, a statement must report something that readers agree not to question, at least for the purposes of the argument. But if they do question it, what you think is hard factual evidence is for them only a reason, and you have not yet reached that bedrock of evidence on which your argument must rest.

I think there is a contract between you and the reader. You must agree to cite sources that are plausibly truthful, and your reader must agree to accept that these sources are reliable. A diligent and well-meaning reader can always second-guess whether, for instance, the beureau of subject matter statistics is collecting and reporting data correctly, but at a certain point this violates the social contract. If they’re genuinely curious or concerned, it may fall on them to investigate the source, not on you. The bar you need to meet is that your sources are plausibly trustworthy. The book doesn’t talk much about this contract, so there’s little I can say about what “plausible” means.

Sometimes you have to be extra careful to distinguish reasons from evidence, a (<claim>, <reason>, <evidence>) tuple is subject to regress in the latter two components, (A, B, C) may need to be justified by (B, C, D) and so on. The example given of this regress is if I told you (american higher education must curb escalating tuition costs, because the price of college is becoming an impediment to the american dream, today a majority of students leave college with a crushing debt burden). In the context of this sentence, “a majority of students...” is evidence, but it would be reasonable to ask for more specifics. In principle, any time information is compressed it may be reasonable to ask for more specifics. A new tuple might look like (the price of college is becoming an impediment to the american dream, because today a majority of students leave college with a crushing debt burden, in 2013 nearly 70% of students borrowed money for college with loans averaging $30000...). The third component is still compressing information, but it’s not in the contract between you and the reader for the reader to demand the raw spreadsheet, so this second tuple might be a reasonable stopping point of the regress.

If you can imagine readers plausibly asking, not once but many times, how do you know that? What facts make it true?, you have not yet reached what readers want—a bedrock of uncontested evidence.

Sometimes you have to be careful to distinguish evidence from reports of it. Again, because we are necessarily dealing with compressed information, we can’t often point directly to evidence. Even a spreadsheet, rather than summary statistics of it, is a compression of the phenomena in base reality that it tracks.

data you take from a source have invariably been shaped by that source, not to misrepresent them, but to put them in a form that serves that source’s ends. … when you in turn report those data as your own evidence, you cannot avoid manipulating them once again, at least by putting them in a new context.

There is a criteria you want to screen your evidence with respect to.

sufficient
representative
accurate
precise
authoritative

Being honest about the reliability and prospective accuracy of evidence is always a positive signal. Evidence can be either too precise or not precise enough. The women in one or two of Shakespeare’s plays do not represent all his women, they are not representative. Figure out what sorts of authority signals are considered credible in your community, and seek to emulate them.

Question your argument as your readers will—thoughts on chapter 10

Three predictable disagreements are

There are causes in addition to the one you claim
What about these counterexamples?
I don’t define X as you do, to me X means...

There are roughly two kinds of queries readers will have about your argument

intrinsic soundness—“challenging the clarity of a claim, relevance of reasons, or quality of evidence”
extrinsic soundness—“different ways of framing the problem, evidence you’ve overlooked, or what others have written on the topic.” The idea is to anticipate, acknowledge, and respond to both kinds of questions. This is the path to making an argument that readers will trust and accept.

Voicing too many hypothetical objections up front can paralyze you. Instead, what you should do before anything else is focus on what you want to say. Give that some structure, some meat, some life. Then, an important exercise is to imagine readers’ responses to it.

I think cleaving these into two highly separated steps is an interesting idea, doing this with intention may be a valuable exercise next time I’m writing something.

View your argument through the eyes of someone who has a stake in a different outcome, someone who wants you to be wrong.

The authors provide some questions about your problem from a possible reader:

Why do you think there’s a problem at all?
Have you properly defined the problem?
Is your solution practical or conceptual?
Have you stated your claim too strongly?
Why is your practical/conceptual solution better than others?

Then, they provide some questions about your support from a possible reader.

“I want to see a different kind of evidence” i.e. hard numbers over anecdotes / real people over cold numbers
“It isn’t accurate”
“It isn’t precise enough”
“It isn’t current”
“It isn’t representative”
“It isn’t authoritative”
“You need more evidence”

It builds credibility to play defense: to recognize your own argument’s limitations. It builds even more credibility to play offense: to explore alternatives to your argument and bring them into your reasoning. If you can, you might develop those alternatives in your own imagination, but more likely you’d like to find alternatives in your sources.

Often your readers will be likeyour sources’ authors; sometimes they may even include them.

What is the perfect amount of objections to acknowledge? Acknowledging too many can distract readers from the core of your argument, while acknowledging too few is a signal of laziness or even disrespect. You need to narrow your list of alternatives or objections by subjecting them to the following priorities

plausible charges of weaknesses that you can rebut
alternative lines of argument important in your field
alternative conclusions that readers want to be true
alternative evidence that readers know
important counterexamples that youu have to address. What if your argument is flawed? The best thing to do is candidly acknowledge the issue and respond that...
the rest of your argument more than balances the flaw
while the flaw is serious, more research will show a way around it
while the flaw makes it impossible to accept your claim fully, your argument offers important insight into the question and suggests what a better answer would need.

It is wise to build up good faith by acknowledging questions you can’t answer. Concessions are often interpreted as positive signals by the reader.

It is important for your responses to acknowledgments to be subordinate to your main point, or else the reader will miss the forest for the trees.

Remember to make an intentional decision about how much credence to give to an objection or alternative. Weaker ones imply weaker credences, imply less effort in your acknowledgment and response.

On warrants—thoughts on chapter 11

This general circumstance ==predictably implies==> this general consequence
    ^                                                       ^
    ||                                                      ||
    ||                                                      ||
is a good instance of                               is a good instance of
    ||                                                      ||
    ||                                                      ||
This specific circumstance ==lets us infer==>     this specific consequence

We saw that arguments take place in five parts. Claims, reasons, evidence, acknowledgments & response, and warrants flow together into into a single argument.

A warrant is just a principle that connects a reason to a claim. Warrants are important because readers may challenge not just the validity of a reason but its relevance as well.

In everyday life, people use proverbs as warrants. If I rush out of the room and leave behind my charging laptop, someone might say “you shouldn’t have rushed”, and if I question the relevance of rushing to leaving behind my laptop they might say “haste makes waste” to illustrate the cause-effect relation. In academic arguments, however, there’s no reliance on commonsense or commonly acknowledged phrases that rhyme, they are instead “specific principles of reasoning that belong to particular communities of researchers”.

Often warrants are implicit, especially in writing by experienced researchers. Like problems, warrants take on the structure of circumstance and consequence.

Warrant-powered arguments proceed in a logic of generalities and instances. A warrant might be “when a powerful system has a goal (general circumstance), the universe gets tiled with counterintuitive realizations of that goal (general consequence)”, which could power an argument like “the atoms in my kidney might get rearranged into smileyfaces (claim) because an arbitrary self-improving system got tasked with making people happy (reason)”. The atoms in my kidney getting rearranged into smileyfaces is a good instance of a counterintuitive realization of a goal, that goal being making people happy. Notice the square above.

When to state a warrant

Warrants can go unsaid quite often. There are roughly three circumstances where you want to be explicit about your warrants.

Your readers are outside your field
You use a principle of reasoning that is new or controversial in your field
You make a claim that readers will resist because they don’t want it to be true

Meta-warrants

It is worth picking out such a thing as methodological warrants or meta-warrants. Think of them as the source of proverbs rather than proverbs themselves. Three basic metawarrants are

Generalization

When every known case of X has quality Y, then all Xs probably have quality Y

Analogy

When X islike Y in most respects, then X will be like Y in other respects

Sign

When Y regularly occurs before, during, or after X, Y is a sign of X

Reasons, evidence, and warrants

You can justify your reasons in two ways: by offering evidence to suport them or by deriving them from a warrant. … Whenever you can, rely not on elaborate lines of reasoning based on warrants but on hard evidence.

Cliffnotes to Craft of Research parts I, II, and III