I work primarily on AI Alignment. Scroll down to my pinned Shortform for an idea of my current work and who I’d like to collaborate with.
Website: https://jacquesthibodeau.com
Twitter: https://twitter.com/JacquesThibs
GitHub: https://github.com/JayThibs
LinkedIn: https://www.linkedin.com/in/jacques-thibodeau/
Yeah, thanks! I agree with @habryka’s comment, though I’m a little worried it may shut down conversation since it might make people think the conversation is about AI startups in general and less about AI startups in service of AI safety. This is because people might consider the debate/question answered after agreeing with the top comment.
That said, I do hear the “any AI startup is bad because it increases AI investment and therefore reduces timelines” so I think it’s worth at least getting more clarity on this.
Three Epoch AI employees* are leaving to co-found an AI startup focused on automating work:
“Mechanize will produce the data and evals necessary for comprehensively automating work.”
They also just released a podcast with Dwarkesh.
*Matthew Barnett, Tamay Besiroglu, Ege Erdil
In case this is useful to anyone in the future: LTFF does not provide funding for-profit organizations. I wasn’t able to find mentions of this online, so I figured I should share.
I was made aware of this after being rejected today for applying to LTFF as a for-profit. We updated them 2 weeks ago on our transition into a non-profit, but it was unfortunately too late, and we’ll need to send a new non-profit application in the next funding round.
FWIW, I was always concerned about people trying to make long-horizon forecast predictions because they assumed superforecasting would extrapolate beyond the sub-1-year predictions that were tested.
As an alternative, that’s why I wrote about strategic foresight to focus on robust plans rather than trying to accurately predict the actual scenario.
We got our first 10k! Woo!
We just put out a Manifund proposal to take short timelines and automating AI safety seriously. I want to make a more detailed post later, but here it is: https://manifund.org/projects/coordinal-research-accelerating-the-research-of-safely-deploying-ai-systems
When is the exact deadline? Is it EOD AOE on February 15th or February 14th? “By February 15th” can sound like the deadline hits as soon as it’s the 15th.
Have seen a few people ask this question in some Slacks.
I keep hearing about dual-use risk concerns when I mention automated AI safety research. Here’s a simple solution that could even work in a startup setting:
Keep all of the infrastructure internally and only share with vetted partners/researchers.
You can hit two birds with one stone:
Does not turn into a mass-market product that leads to dual-use risks.
Builds a moat where you have complex internal infrastructure which is not shared, only the product of that system is shared. Investors love moats, you just got to convince them that this is the way to go for a product like this these days.
You don’t market the product to mass-market, you just find partners and use the system to spin out products and businesses that have nothing to do with frontier models. So, you can repurpose the system for specific application areas without releasing the platform and process, which would be copied in a day in the age of AI anyways.
I’m currently working on de-slopifying and building an AI safety startup with this as a central pillar.* Happy to talk privately with anyone working on AI safety who is interested in this.
*almost included John and Gwern’s posts on AI slop as part of a recent VC pitch deck.
I’m working on this. I’m unsure if I should be sharing what I’m exactly working on with a frontier AGI lab though. How can we be sure this just leads to differentially accelerating alignment?
Edit: my main consideration is when I should start mentioning details. As in, should I wait until I’ve made progress on alignment internally before sharing with an AGI lab. Not sure what people are disagreeing with since I didn’t make a statement.
I’m currently in the Catalyze Impact AI safety incubator program. I’m working on creating infrastructure for automating AI safety research. This startup is attempting to fill a gap in the alignment ecosystem and looking to build with the expectation of under 3 years left to automated AI R&D. This is my short timelines plan.
I’m looking to talk (for feedback) to anyone interested in the following:
AI control
Automating math to tackle problems as described in Davidad’s Safeguarded AI programme.
How to robustify society in a post-AGI world
Leverage large amounts of inference-time compute to make progress on alignment research
Short timelines
Profitability while still reducing overall x-risk
Are someone with an entrepreneurial spirit and can spin out traditional business within the org to fund the rest of the work (thereby reducing investor pressure)
If you’re interested in chatting or giving feedback, please DM me!
Are you or someone you know:
1) great at building (software) companies
2) care deeply about AI safety
3) open to talk about an opportunity to work together on something
If so, please DM with your background. If someone comes to mind, also DM. I am looking thinking of a way to build companies in a way to fund AI safety work.
In case you didn’t read Paul’s reasoning.
Agreed, but I will find a way.
Hey Ben and Jesse!
This comment is more of a PSA:
I am building a startup focused on making this kind of thing exceptionally easy for AI safety researchers. I’ve been working as an AI safety researcher for a few years. I’ve been building an initial prototype and I am in the process of integrating it easily into AI research workflows. So, with respect to this post, I’ve been actively working towards building a prototype for the “AI research fleets”.
I am actively looking for a CTO I can build with to +10x alignment research in the next 2 years. I’m looking for someone absolutely cracked and it’s fine if they already have a job (I’ll give my pitch and let them decide).
If that’s you or you know anyone who could fill that role (or who I could talk to that might know), then please let me know!
For alignment researchers or people in AI safety research orgs: hit me up if you want to be pinged for beta testing when things are ready.
For orgs, I’d be happy to work with you to setup automations or give a masterclass on the latest AI tools/automation workflows and maybe provide a custom report (with a video overview) each month so that you can focus on research rather than trying new tools that might not be relevant to your org.
Additional context:
“When we say “automating alignment research,” we mean a mix of Sakana AI’s AI scientist (specialized for alignment), Transluce’s work on using AI agents for alignment research, test-time compute scaling, and research into using LLMs for coming up with novel AI safety ideas. This kind of work includes empirical alignment (interpretability, unlearning, evals) and conceptual alignment research (agent foundations).
We believe that it is now the right time to take on this project and build this startup because we are nearing the point where AIs could automate parts of research and may be able to do so sooner with the right infrastructure, data, etc.
We intend to study how our organization’s work can integrate with the Safeguarded AI thesis by Davidad.”
I’m currently in London for the month as part of the Catalyze Impact programme.
If interested, send me a message on LessWrong or X or email (thibo.jacques @ gmail dot com).
It has basically significantly accelerated my ability to build fully functional websites very quickly. To the point where it was basically a phase transition between me building my org’s website and not building it (waiting for someone with web dev experience to do it for me).
I started my website by leveraging the free codebase template he provides on his github and covers in the course.
I mean that it’s a trade secret for what I’m personally building, and I would also rather people don’t just use it freely for advancing frontier capabilities research.
Is this because it would reveal private/trade-secret information, or is this for another reason?
Yes (all of the above)
I can’t think of anyone making a call worded like that. The closest I can think of is Christiano mentioning, in a 2023 talk on how misalignment could lead to AI takeover, that we’re pretty close to AIs doing things like reward hacking and threatening users, and that he doesn’t think we’d shut down this whole LLM thing even if that were the case. He also mentioned we’ll probably see some examples in the wild, not just internally.