Having known some of Conjecture’s founders and their previous work in the context of “early-stage EleutherAI”, I share some[1] of the main frustrations outlined in this post. At the organizational level, even setting aside the departure of key researchers, I do not think that Conjecture’s existing public-facing research artifacts have given much basis for me to recommend the organization to others (aside from existing personal ties). To date, only[2] a few posts like their one on the polytope lens and their one on circumventing interpretability were at the level of quality & novelty I expected from the team. Maybe that is a function of the restrictive information policies, maybe a function of startup issues, maybe just the difficulty of research. In any case, I think that folks ought to require more rigor and critical engagement from their future research outputs[3].
I didn’t find the critiques of Connor’s “character and trustworthiness” convincing, but I already consider him a colleague & a friend, so external judgments like these don’t move the needle for me.
The main other post I have in mind was their one on simulators. AFAICT the core of “simulator theory” predated (mid-2021, at least) Conjecture, and yet even with a year of additional incubation, the framework was not brought to a sufficient level of technical quality.
For example, the “cognitive emulation” work may benefit from review by outside experts, since the nominal goal seems to be to do cognitive science entirely inside of Conjecture.
Having known some of Conjecture’s founders and their previous work in the context of “early-stage EleutherAI”, I share some[1] of the main frustrations outlined in this post. At the organizational level, even setting aside the departure of key researchers, I do not think that Conjecture’s existing public-facing research artifacts have given much basis for me to recommend the organization to others (aside from existing personal ties). To date, only[2] a few posts like their one on the polytope lens and their one on circumventing interpretability were at the level of quality & novelty I expected from the team. Maybe that is a function of the restrictive information policies, maybe a function of startup issues, maybe just the difficulty of research. In any case, I think that folks ought to require more rigor and critical engagement from their future research outputs[3].
I didn’t find the critiques of Connor’s “character and trustworthiness” convincing, but I already consider him a colleague & a friend, so external judgments like these don’t move the needle for me.
The main other post I have in mind was their one on simulators. AFAICT the core of “simulator theory” predated (mid-2021, at least) Conjecture, and yet even with a year of additional incubation, the framework was not brought to a sufficient level of technical quality.
For example, the “cognitive emulation” work may benefit from review by outside experts, since the nominal goal seems to be to do cognitive science entirely inside of Conjecture.