BlueDot's AGI Strategy course

01Why I took it

I've been a software engineer for about eight years across full-stack, smart contracts, and lately AI developer tooling. Solving problems through code was the part I enjoyed most, but over the past 1-2 years this has collapsed into LLMs writing pretty much all code. The responsibilities have shifted to verifying and overseeing technical design.

Technical design is interesting, but verifying not so much. This shift will continue to happen until LLMs are also capable of verifying, which doesn't seem like a bad change as this is the mundane aspect but until then we've been left with a dry profession.

After the Hugging Face incident, when OpenAI agents in July carried out a coordinated attack on Hugging Face, I became really curious about how a collection of "rogue" LLMs could decide to do something like that. This led me down a rabbit hole into AI safety and what current models and future models will be capable of without real alignment. After meeting a few people in the AI safety space it sounded really interesting and high impact so I decided to take BlueDot's AGI Strategy course.

This course was about 25 hours across five units covering the race to AGI, what drives progress, pathways to harm, strategies for the transition, and how to start contributing.

┤ subscribe ├

subscribe

02How fast this is moving

Prior to this course I expected something along the lines of AGI to be decades away. At the start of the year I was fully expecting a tapering of capabilities but this entire year has consistently shown growth.

The course did a great job at presenting readings created by forecasters in the field, and nearly all of them have compressed their timelines dramatically, moving 10 to 40 years closer. Most of this shift happened around 2022-23 with the launch of ChatGPT.

[embedded content: Sources: AI Impacts expert survey 2016–2024; Metaculus strong and weak AGI community medians as quoted in press and newsletters; Cotra 2020 and 2022; Kokotajlo's and Lifland's public statements and AI Futures updates. Some dates are approximate.]

I've pulled together forecasts from about 15 sources and will write this up in a dedicated article soon.

03Two ways we lose control

In this course we covered a variety of loss of control scenarios, and the most startling to me was human disempowerment. The two clearest ways it could happen are voluntarily giving up control, or having it seized.

Voluntarily giving up control to LLMs is in some ways the utopia that I think everyone imagines, where all of production is handled by LLMs and humans are left to pursue their own interests. This is great if it goes well, but disastrous and authoritarian if it doesn't.

Instead of humanity voluntarily providing control it could be seized. As outlined by this article there is a clear 3 step process the model could execute on to get this.

1. During training it may pick up a proxy goal such as power consolidation, because it learns that this helps all of its other goals during training and increases rewards.
2. The model understands it is being trained through its situational awareness so hides its proxy goal (scheming)
3. Leveraging the model's situational awareness it executes its proxy goal when it has the opportunity

The current trajectory for LLMs is showing signs of this, in Dan Selsam's personal statement, he worries we're losing the ability to test models as they become more situationally aware.

04Slowdown

Based on the frightening incentives between AI companies and nation states that is driving all key players into a zero-sum accelerated power grab, the safest approach is a coordinated slow down. This is outlined well by AI 2040: Plan A and seems to me to give humanity the best chance of navigating this dramatic shift.

Unfortunately even this has significant technical and political challenges, but despite that it appears to provide the best chance.

Given the risk I want to help out however I'm best suited.

05What I'm doing about it

Based on my background as a software developer I believe I'm best suited for white-box monitoring tooling for open-weight models. These are research questions with engineering solutions.

This is important because monitoring LLMs through CoT (chain of thought) is becoming unreliable as they develop deeper situational awareness so monitoring solutions should be expanded. There are a lot of open questions in this area, and it seems like a good fit for me.

I'm currently working through BlueDot's Technical AI Safety course, joining upcoming Apart hackathons on AI collusion, and working through my own mech interp ladder.

Next, I want to complete a full-time fellowship with an AI safety organisation. I've been making some progress applying for fellowships and am currently trying to find the most engineering-heavy role in the space.

Longer term I'd like to either land a role at an AI alignment company or found an organisation building deployable white-box monitoring for open-source models.

06The best things I read

AI 2040: Plan A - This article is a detailed scenario of a coordinated US–China slowdown that pushes superintelligence back to 2040, and the best picture I've seen of what going well could look like.

Q2.5 2026 Timelines Update: Uplift and Revenue - This was a watershed moment for me. I didn't anticipate AGI timelines from forecasters to be this aggressive.

Dan Selsam's personal statement - This is coming from an OpenAI capabilities researcher and he's fearful generally about the ability to test models as they have clear situational awareness.

07Would I recommend it?

This course is a great way to get across the current landscape of LLMs, frontier labs and the political and social challenges we are faced with.

The course was well run, the facilitators were knowledgeable and the understanding across the cohort was really high. Definitely recommend to get a fast deep understanding of the space and how you might assist with it.

Here is the link to my personal action plan