(Photo by Glenn Carstens-Peters on Unsplash)
In an effort to advance AI safety, I just spent 30 days contributing to Inspect AI, an open-source tool for AI model evaluations. I hope my contributions make it easier for people to work with and work on Inspect so we can better evaluate risks in AI models.
When I started, I thought that surely, in 30 days, I’d fix a bug or two, or maybe even add a new feature. Instead, I wrote documentation. Which is exactly what the project needed and what I was uniquely positioned to offer. Over those 30 days, I opened three issues, merged five pull requests, and directly contributed to closing three additional issues.
Here’s what I learned over the course of 30 days and in my reflection afterwards. If you’re going to commit to a 30-day plan like this, here’s what I’d recommend based on what worked – and what didn’t – for me:
Choose one: make a meaningful impact or learn a valuable skill
Either choose a project where you can (a) make a meaningful impact or (b) learn a valuable new skill. I chose to contribute to a tool that’s used by some of the research orgs in the AI safety space since I wanted to be able to help them by improving their tooling (to make a meaningful impact) and continue developing my Python skills (to learn something new and valuable).
I discovered that trying to do both was a mistake.
I was able to make a meaningful impact, but I didn’t spend much time with Python since my time was limited. Given enough time, I could learn more Python while contributing meaningfully to Inspect, but that wasn’t realistic for me to do in an hour a day over 30 days. I’d need more time to be able to achieve both goals. I’d also need to find a “good first issue” for Inspect, but unfortunately there were none labeled as such, and I wasn’t able to find any on my own.
So my advice: pick one. Focus on either making a meaningful impact or learning a valuable new skill. You can do the other in your next 30 days.
Give yourself an achievable challenge
I had the unrealistic goal of fixing a bug or adding a new feature to Inspect. It just wasn’t going to happen with the time I had. Here’s what I said in my plan:
I’m going to work on Inspect AI through open-source contributions, fixing bugs and adding new features as needed. I’ll fix at least one issue over the next 30 days, contributing my fix via a pull request.
In retrospect, “fixing bugs and adding new features” was wildly optimistic.
Fixing one documentation issue, though? That was easy.
When you’re new to a project – like I was to Inspect – you’re uniquely positioned to point out where the gaps are in existing docs, especially around tutorials, processes, and how to use the tools. You’ll get stuck on things, and that’s an incredibly valuable signal – you’ve found a gap in the documentation. Report an issue to let the maintainers know, and boom, you’ve made your first contribution.
After I accomplished my goal of fixing one issue in the first week, I was left wondering what to do next. I decided to just keep going, fixing more documentation issues. It would have been nice if, at the outset, I’d set some additional goals for myself. My plan could have been to fix at least three documentation issues, with a stretch goal of fixing five.
Giving yourself an achievable challenge is tough – especially on a project that’s new to you – because what you can do depends on the state of the project, and you don’t really know the state. Maybe there are some glaring weaknesses in the docs to be addressed, or some obvious bugs to fix, in which case you might want to try fixing a few. Or maybe there are only big issues and fixing just one will take you the entire 30 days.
It’s hard to know what will be achievable and challenging on a new project. Start with your best guess, then make adjustments after you’re a week or two into it.
Break it down
I broke down my 30 days into the following actions:
- Comment on the issue titled “Improve display of grouped metrics” and ask if it’s been completed. Deadline: Monday, Nov 10.
- Get more familiar with Inspect by writing and running evals. Write and run at least one by Wednesday, Nov 12.
- Choose an issue that I can fix, work on it, and contribute my fix (via an open pull request). Deadline: Wednesday, Dec 10.
The problem with this plan is that there’s nothing between Nov 12 and Dec 10. This is too long for me to spend on my first issue on a project. I could have broken it down more, perhaps with one thing to do each week. If you’re making a 30-day plan, I’d suggest something like this:
- Week 1: Follow the tutorial or README that explains how to use the tool. Read the Contributing guide. When you find problems in any of the docs, open an issue.
- Week 2: Set up your dev environment. Build the project and run the tests. Fix your own documentation issue and submit a pull request.
- Week 3–4: Look for small code issues you can fix (“good first issue”, perhaps) or continue to improve existing docs.
- Stretch: Fix a bigger issue or feature.
Get a friend or two to join you
My 30-Day Personal Action Plan was the capstone of BlueDot’s Technical AI Safety course, which I was lucky enough to take with a cohort. In our final discussion where we presented our plans, we also decided to reconnect at the end of the 30 days. For me, this was motivating – I wanted to be able to tell everyone that I’d accomplished my goals. And reconnecting with a small group at the end of the 30 days to discuss and reflect on how it went was both fun and insightful. I’m proud of us for moving AI safety forward, even if we didn’t completely follow our plans or accomplish all our goals.
Good luck!
All the best in accomplishing your goals for your 30-Day Personal Action Plan!
You can see my Inspect contributions on GitHub. I’m continuing to contribute to AI safety tooling and writing about the process. If this was valuable, you can support this work on GitHub Sponsors.