My top interest is AI safety, followed by reinforcement learning. My professional background is in software engineering, computer science, machine learning. I have degrees in electrical engineering, liberal arts, and public policy. I currently live in the Washington, DC metro area; before that, I lived in Berkeley for about five years.
David James
Karma: 123
Tools for decision-support, deliberation, sense-making, reasoning
I’m putting many of these in a playlist along with The Geeks Were Right by The Faint: https://www.youtube.com/watch?v=TF297rN_8OY
When I saw the future—the geeks were right
Egghead boys with thin white legs
They’ve got modified features and software brains
But that’s what the girls like—the geeks were rightPredator skills, chemical wars, plastic islands at sea
Watch what the humans ruin with machines
“If you see fraud and do not say fraud, you are a fraud.”—Nasim Taleb
No. Taleb’s quote is too simplistic. There is a difference between (1) committing fraud; (2) denying fraud where it exists; and (3) saying nothing.
Worse, it skips over a key component of fraud: intent!
I prefer the following framing: If a person sees evidence of fraud, they should reflect on (a) the probability of fraud (which involves assessing the intention to deceive!); (b) their range of responses; (c) the effects of each response; and (d) what this means for their overall moral assessment.
I realize my framing draws upon consequentialist reasoning, but I think many other ethical framings would still criticize Taleb’s claim for being overly simplistic.
The comment above may open a Flood of Jesus-Backed Securities and Jesus-Leveraged Loans. Heavens!
The recent rise of reinforcement learning (RL) for language models introduces an interesting dynamic to this problem.
Saying “recent rise” feels wrong to me. In any case, it is vague. Better to state the details. What do you consider to be the first LLM? The first use of RLHF with a LLM? My answers would probably be 2018 (BERT) and 2019 (OpenAI), respectively.
HLE and benchmarks like it are cool, but they fail to test the major deficits of language models, like how they can only remember things by writing them down onto a scratchpad like the memento guy.
A scratch pad for thinking, in my view, is hardly a deficit at all! Quite the opposite. In the case of people, some level of conscious reflection is important and probably necessary for higher-level thought. To clarify, I am not saying consciousness itself is in play here. I’m saying some feedback loop is probably necessary — where the artifacts of thinking, reasoning, or dialogue can themselves become objects of analysis.
My claim might be better stated this way: if we want an agent to do sufficiently well on higher-level reasoning tasks, it is probably necessary for them to operate at various levels of abstraction, and we shouldn’t be surprised if this is accomplished by way of observable artifacts used to bridge different layers. Whether the mechanism is something akin to chain of thought or something else seems incidental to the question of intelligence (by which I mean assessing an agent’s competence at a task, which follows Stuart Russell’s definition).
I don’t think the author would disagree, but this leaves me wondering why they wrote the last part of the sentence above. What am I missing?
A just world is a world where no child is born predetermined to endure avoidable illness simply because of ancestral bad luck.
In clear-cut cases, this principle seems sound; if a certain gene only has deleterious effects, and it can be removed, this is clearly better (for the individual and almost certainly for everyone else too).
In practice, this becomes more complicated if one gene has multiple effects. (This may occur on its own or because the gene interacts with other genes.) What if the gene in question is a mixed bag? For example, consider a gene giving a 1% increased risk of diabetes while always improving visual acuity. To be clear, I’m saying complicated not unresolvable. Such tradeoffs can indeed be resolved with a suitable moral philosophy combined with sufficient data. However, the difference is especially salient because the person deciding isn’t the person that has to live with said genes. The two people may have different philosophies, risk preferences, or lifestyles.
Not necessarily an optimizer, though: satisficers may do it too. A core piece often involves tradeoffs, such as material efficiency versus time efficiency.
A concrete idea: what if every LessWrong article prominently linked to a summary? Or a small number of highly-ranked summaries? This could reduce the burden on the original author, at the risk of having the second author’s POV differ somewhat.
What if LW went so far as to make summaries the preferred entry ways? Instead of a reader seeing a wall of text, they see a digestible chunk first?
I have been wanting this for a very long time. It isn’t easy nor obvious nor without hard trade-offs. In any case, I don’t know of many online forums nor information sources that really explore the potential here.
Related: why not also include metadata for retractions, corrections, and the like? TurnTrout’s new web site, for example, sometimes uses “info boxes” to say things like “I no longer stand by this line of research”.
At least when I’m reading I like to have some filler between the ideas to give me time to digest a thought and get to the next one.
This both fascinating and strange to me.
If you mean examples, elaboration, and explanation, then, yes, I get what you mean.
OTOH, if you mean “give the reader a mental break”, that invites other alternatives. For example, if you want to encourage people to pause after some text, it might be worthwhile to make it harder to mindlessly jump ahead. Break the flow. This can be done in many ways: vertical space, interactive elements, splitting across pages, and more.
This is a fun design space. So much about reading has evolved over time, with the medium imposing constraints on the process. We have more feasible options now!
and I don’t really see how to do that without directly engaging with the knowledge of the failure modes there.
I agree. To put it another way, even if all training data was scrubbed of all flavors of deception, how could ignorance of it be durable?
If you have a clear policy objective, you can probably find someone, somewhere to give you a fair hearing.
To clarify, are you suggesting now is a better time than, say one year ago? If so, here are some factors working against such a claim: (a) There are fewer people around, so reaching someone is going to be harder. (b) The people that remain are trying to survive, which involves keeping a low profile. (c) People that will hear you out feel immense pressure to tow the line, which is usually considered the opposite of entertaining new ideas. (d) If an idea gets some traction, any sensible staffer will wonder what chaos will emerge next to render the idea untenable.
Now, if you happen to get an audience for a policy idea, it is also important to ask yourself (i) What is the experience level of the staffer in front of you? (ii) Do they understand how the system works? (iii) Will they be effective stewards for your policy goal?
In this climate especially, one cannot ignore concerns about stability and corruption. The leaders of the current administration seek to expand the power of the executive branch significantly. They are willing to stretch—and break—the rule of law, as various court orders have demonstrated. My point? An unstable political and legal environment is not conducive to serious policy aims. Policy, no matter what the aim, is predicated on a legal foundation that operates over time in some kind of known environment.
For example, if one’s actual policy objective is to, say, modernize the IRS (which I would support, if done properly), there are steps to do this. Given the Republican Party’s control of all three branches of government, they could do this legally. Many (perhaps most?) rational thinkers would support simplifying the tax code, increasing compliance, and increasing operational efficiency, even though we have different ideas about the aims and scope of government policy.
Academia is less selective than it used to be
To what degree is this true regarding elite-level Ph.D. programs that are likely to lead to publication in (i) mathematics and/or (ii) computer science?
Separately, we should remember that academic selection is a relative metric, i.e. graded on a curve. So, when it comes to Ph.D. programs, is the median 2024 Ph.D. graduate more capable (however you want to define it) than the corresponding graduate from 1985? This is complex, involving their intellectual foundations, depth of their specialized knowledge, various forms of raw intelligence, attention span, collaborative skills, communication ability (including writing skills), and computational tools?
I realize what I’m about to say next may not be representative of the median Ph.D. student, but it feels to me the 2024 graduates of, say, Berkeley or MIT (not to mention, say, Thomas Jefferson High School) are significantly more capable than the corresponding 1985 graduates. Does my sentiment resonate with others and/or correspond to some objective metrics?
For me, I’d say a lot of my gains come from asking AI questions rather than generating code directly.
This is often the case for me as well. I often work on solo side projects and use Claude to think out loud. This lets me put on different hats, just like when pair programming, including: design mode, implementation mode, testing mode, and documentation mode.
I rarely use generated code as-is, but I do find it interesting to look at. As a concrete example, I recently implemented a game engine for the board game Azul (and multithreaded solver engine) in Rust and found Claude very helpful for being an extra set of eyes. I used it sort of a running issue tracker, design partner, and critic.
Now that I think about it, maybe the best metaphor I can use is that Claude helps me project myself onto myself. For many of my projects, I lean towards “write good, understandable code” instead of “move fast and break things”. This level of self-criticism and curiosity has served me well with Claude. Without this mentality, I can see why people dismiss LLM-assisted coding; it certainly is far from a magic genie.
I’ve long had a bias toward design-driven work (write the README first, think on a whiteboard, etc), whether it be coding or almost anything, so having an infinitely patient conversational partner can be really amazing at times. At other times, the failure modes are frustrating, to say the least.
I agree that having a binder of policy proposals ready is effective. There is a dark side to this too. If you are a policy maker, expect plenty of pre-prepared binders awaiting the situation you find yourself in. Different groups vary widely in their predictive abilities and intellectual honesty.
The history of think tanks is fascinating and complicated. On one hand, they provide a bridge from academia to policy that can retain some of the intellectual rigor of the former. On the other hand, they can be thinly veiled ideologically motivated places awaiting a favorable political environment.
Right. To expand on this: there are also situations where an interest group pushes hard on a broader coalition to move faster, sometimes even accusing their partners or allies of “not caring enough” or “dragging their feet”. Assuming bad faith or impugning the motives of one’s allies can sour working relationships. Understanding the constraints in play goes a long way towards fostering compromise.
The idea of “focusing events” is well known in public policy.
For example, see Thomas Birkland’s book “After Disaster: Agenda Setting, Public Policy, and Focusing Events” or any of his many articles such as “Focusing Events, Mobilization, and Agenda Setting” (Journal of Public Policy; Vol. 18, No. 1; 1998)
According to Birkland in “During Disaster: Refining the Concept of Focusing Events to Better Explain Long-Duration Crises”, John Kingdon first used the term “focusing events” in his book “Agendas, Alternatives, and Public Policy”.
There is a considerable literature on these topics that does not rely on Milton Friedman or his political philosophy. Invoking Friedman in policy circles can make it harder to have neutral conversation about topics unrelated to markets, such as the Overton window and “theories of change” which thankfully seem to have survived as both neutral and intellectually honest ways of talking about the policy process.
With this in mind, I suggest listing these other authors alongside Milton Friedman to give a broader context. This will help us flawed humans focus on the core ideas rather than wonder in the back of our heads if the ideas are part of a particular political philosophy. As such, it probably will help to get these concepts in wider circulation.
To the students of history out there, let me know to what degree Friedman played a key role in developing and/or socializing the ideas around crises and focusing events. If so, credit where credit it due.
For what it is worth, Friedman, Arthur Okun (“Equality and Efficiency”), and Birkland were assigned reading in my public policy studies. We were expected to be able to articulate all of their points of view clearly and honestly, even if we disagreed.
I find this article confusing. So I find myself returning to fundamentals of computer science algorithms: to greedy algorithms and under what conditions they are optimal. Would anyone care to build a bridge from this terminology to what the author is trying to convey?
I wonder if you underestimate the complexity of brokering, much less maintaining, a lasting peace, whether it be via superior persuasive abilities or vast economic resource advantages. If you are thinking more along the lines of domination that is so complete that any violent resistance seems minuscule and pointless that’s a different category for me. When I think of “long term peace”, I usually don’t think of simmering grudges that remain dormant because of a massive power imbalance. I will grant that perhaps ultimate form of “persuasion” would involve removing even the mental possibility of resistance.
I can see the appeal here—litanies tend to have a particular style after all—but I wonder if we can improve it.
I see two problems:
This doesn’t convey that Occam’s razor is about explanations of observations.
In general, one explanation is not a logical “subset” of the other. So the comparison is not between
AandA and B; it is betweenAandB.Perhaps one way forward would involve a mention (or reference to) Minimum Description Length (MDL) or Kolmogorov complexity.