Geoffrey Irving
@girving.bsky.social
Chief Scientist at the UK AI Security Institute (AISI). Previously DeepMind, OpenAI, Google Brain, etc.
The one silver lining is that I'm originally from Alaska, so sometimes I get to refer to just "40 below" with no qualifiers.
November 28, 2025 at 10:31 AM
The one silver lining is that I'm originally from Alaska, so sometimes I get to refer to just "40 below" with no qualifiers.
It's even more confusing when I'm an American living in London.
November 28, 2025 at 10:31 AM
It's even more confusing when I'm an American living in London.
Please reach out if you're interested in co-funding. We'd love to close this gap and further grow the field of alignment research! ❤️
November 27, 2025 at 6:25 PM
Please reach out if you're interested in co-funding. We'd love to close this gap and further grow the field of alignment research! ❤️
We've already expanded the initial budget of £15M significantly, but there's still a £13-17M gap to fund all shortlisted, high-potential projects. I would love to close the gap, both to ensure the excellent projects happen and to cement alignment as a field people can pivot into.
November 27, 2025 at 6:25 PM
We've already expanded the initial budget of £15M significantly, but there's still a £13-17M gap to fund all shortlisted, high-potential projects. I would love to close the gap, both to ensure the excellent projects happen and to cement alignment as a field people can pivot into.
Many proposals have the potential to find *definitions and problem reframings*, which are particularly important as they can help with further progress, field building, and other interactions between government and alignment (through mapping the relationship to economics).
November 27, 2025 at 6:25 PM
Many proposals have the potential to find *definitions and problem reframings*, which are particularly important as they can help with further progress, field building, and other interactions between government and alignment (through mapping the relationship to economics).
Potential grantees include some of the world's top complexity theorists proposing obfuscated arguments work (multiple proposals) and theoretical economists reframing the problem in bounded rationality language which may bring additional tools to bear (multiple proposals).
November 27, 2025 at 6:25 PM
Potential grantees include some of the world's top complexity theorists proposing obfuscated arguments work (multiple proposals) and theoretical economists reframing the problem in bounded rationality language which may bring additional tools to bear (multiple proposals).
Many of the proposals are very creative and counterfactual: bringing in new ideas including about many-year-old problems in alignment. To pick out one good example: the obfuscated arguments problem, a core issue with scalable oversight found by Beth Barnes in 2020.
November 27, 2025 at 6:25 PM
Many of the proposals are very creative and counterfactual: bringing in new ideas including about many-year-old problems in alignment. To pick out one good example: the obfuscated arguments problem, a core issue with scalable oversight found by Beth Barnes in 2020.
I think this is a testament to
1. The desire of researchers to enter the field, bringing new ideas and techniques with them.
2. The work by our Alignment Team to scope out research areas, and to actively solicit and advise as people pivot to alignment.
alignmentproject.aisi.gov.uk/research-age...
1. The desire of researchers to enter the field, bringing new ideas and techniques with them.
2. The work by our Alignment Team to scope out research areas, and to actively solicit and advise as people pivot to alignment.
alignmentproject.aisi.gov.uk/research-age...
Research Agenda — Alignment Project by AISI
See the Research Agenda for AISI's Alignment Project.
alignmentproject.aisi.gov.uk
November 27, 2025 at 6:25 PM
I think this is a testament to
1. The desire of researchers to enter the field, bringing new ideas and techniques with them.
2. The work by our Alignment Team to scope out research areas, and to actively solicit and advise as people pivot to alignment.
alignmentproject.aisi.gov.uk/research-age...
1. The desire of researchers to enter the field, bringing new ideas and techniques with them.
2. The work by our Alignment Team to scope out research areas, and to actively solicit and advise as people pivot to alignment.
alignmentproject.aisi.gov.uk/research-age...
The theory of impact of the project is that we could find a bunch of world-class researchers with relevant expertise who haven't worked on alignment, and help them pivot into the field. The number of quality of applications we received is great evidence for this!
November 27, 2025 at 6:25 PM
The theory of impact of the project is that we could find a bunch of world-class researchers with relevant expertise who haven't worked on alignment, and help them pivot into the field. The number of quality of applications we received is great evidence for this!
And yet, there are surprisingly few useful theorems whose natural domain is the primes which aren't 47. :)
November 23, 2025 at 6:24 PM
And yet, there are surprisingly few useful theorems whose natural domain is the primes which aren't 47. :)
More details in the AISI post here: x.com/AISecurityIn...
AI Security Institute on X: "Last week, we hosted our inaugural Alignment Conference, in partnership with @farairesearch. The event bought together an interdisciplinary delegation of leading researchers, funders, and policymakers to discuss urgent open problems in AI alignment 🧵 https://t.co/3L3SWV8Dbj" / X
Last week, we hosted our inaugural Alignment Conference, in partnership with @farairesearch. The event bought together an interdisciplinary delegation of leading researchers, funders, and policymakers to discuss urgent open problems in AI alignment 🧵 https://t.co/3L3SWV8Dbj
x.com
November 13, 2025 at 5:00 PM
More details in the AISI post here: x.com/AISecurityIn...
Thanks to everyone involved in pulling off the conference! FAR.AI and Vael Gates for organisation, the AISI Alignment Team for tons of work on technical content, attendee outreach, and conference delivery, and SAIF, Halcyon Futures, and the AI Security Institute for funding.
November 13, 2025 at 5:00 PM
Thanks to everyone involved in pulling off the conference! FAR.AI and Vael Gates for organisation, the AISI Alignment Team for tons of work on technical content, attendee outreach, and conference delivery, and SAIF, Halcyon Futures, and the AI Security Institute for funding.
There were a bunch of fields represented, but I was specifically happy with the theoretical economics attendees: there's both a ton of game theory potentially relevant to alignment and a bunch of modelling right about how AI will integrate into the world.
November 13, 2025 at 5:00 PM
There were a bunch of fields represented, but I was specifically happy with the theoretical economics attendees: there's both a ton of game theory potentially relevant to alignment and a bunch of modelling right about how AI will integrate into the world.
And I’ve thought about alignment a bunch: a much larger fraction of ideas were novel for the attendees. This is what happens when we pull new people into the field!
November 13, 2025 at 5:00 PM
And I’ve thought about alignment a bunch: a much larger fraction of ideas were novel for the attendees. This is what happens when we pull new people into the field!
I had great conversations with lots of people, and came away with a long reading list and a variety of ideas that hadn’t occurred to me before (new complexity theory tools, the idea of mixing reputation with learning dynamics, epinet/SGLD interactions, a UI variant on debate).
November 13, 2025 at 5:00 PM
I had great conversations with lots of people, and came away with a long reading list and a variety of ideas that hadn’t occurred to me before (new complexity theory tools, the idea of mixing reputation with learning dynamics, epinet/SGLD interactions, a UI variant on debate).
That worry was super fake: we got tons of good proposals. (Important to say that the bar is not solving the problem, but proposing new tools and methods that have a shot at making progress.)
But we still ran the conference, and I am super happy with how it went!
But we still ran the conference, and I am super happy with how it went!
November 13, 2025 at 5:00 PM
That worry was super fake: we got tons of good proposals. (Important to say that the bar is not solving the problem, but proposing new tools and methods that have a shot at making progress.)
But we still ran the conference, and I am super happy with how it went!
But we still ran the conference, and I am super happy with how it went!
We planned the conference as part of the Alignment Project, the goal of which was to get academics and third-party researchers with relevant expertise to shift towards alignment research, in case we didn't get enough good proposals at first.
bsky.app/profile/girv...
bsky.app/profile/girv...
I am very excited that AISI is announcing over £15M in funding for AI alignment and control, in partnership with other governments, industry, VCs, and philanthropists!
Here is a 🧵 about why it is important to bring more independent ideas and expertise into this space.
alignmentproject.aisi.gov.uk
Here is a 🧵 about why it is important to bring more independent ideas and expertise into this space.
alignmentproject.aisi.gov.uk
The Alignment Project by AISI — The AI Security Institute
The Alignment Project funds groundbreaking AI alignment research to address one of AI’s most urgent challenges: ensuring advanced systems act predictably, safely, and for society’s benefit.
alignmentproject.aisi.gov.uk
November 13, 2025 at 5:00 PM
We planned the conference as part of the Alignment Project, the goal of which was to get academics and third-party researchers with relevant expertise to shift towards alignment research, in case we didn't get enough good proposals at first.
bsky.app/profile/girv...
bsky.app/profile/girv...
Do they not have momentum? Presumably detected flaws make it more likely more flaws will be detected.
November 12, 2025 at 1:44 PM
Do they not have momentum? Presumably detected flaws make it more likely more flaws will be detected.