BenjMurrell
@benjmurrell.bsky.social
🇸🇪🇿🇦 Researcher at Karolinska. Comp bio. Phylogenetics. Deep learning. All in Julia. Virology. Immunology.
https://scholar.google.com/citations?user=I80vy5cAAAAJ
https://scholar.google.com/citations?user=I80vy5cAAAAJ
A technical thread on loss scaling in diffusion and flow matching models (related to a new preprint):
Since the dawn of time, people have been messing with (or dropping entirely) these pesky time-dependent loss scaling terms, mostly because the models train better without them.
Since the dawn of time, people have been messing with (or dropping entirely) these pesky time-dependent loss scaling terms, mostly because the models train better without them.
November 21, 2025 at 11:09 PM
A technical thread on loss scaling in diffusion and flow matching models (related to a new preprint):
Since the dawn of time, people have been messing with (or dropping entirely) these pesky time-dependent loss scaling terms, mostly because the models train better without them.
Since the dawn of time, people have been messing with (or dropping entirely) these pesky time-dependent loss scaling terms, mostly because the models train better without them.
With my wonderful lab, who mostly aren't on here (except @lukasbillera.bsky.social and @antonoresten.bsky.social ?) we've been tinkering in this space since the end of the summer, but we think this is just too cool to sit on any longer.
The manuscript should be up by tomorrow and I'll drop a link.
The manuscript should be up by tomorrow and I'll drop a link.
November 10, 2025 at 9:10 AM
With my wonderful lab, who mostly aren't on here (except @lukasbillera.bsky.social and @antonoresten.bsky.social ?) we've been tinkering in this space since the end of the summer, but we think this is just too cool to sit on any longer.
The manuscript should be up by tomorrow and I'll drop a link.
The manuscript should be up by tomorrow and I'll drop a link.
Eg. here is a protein example where the model designs two domains with an intervening linker (light blue chain in vid). A regular flow model would need to know, early on, exactly how many AAs are needed in the linker, but Branching Flows can decide on-the-fly and grow or shrink it.
November 10, 2025 at 9:10 AM
Eg. here is a protein example where the model designs two domains with an intervening linker (light blue chain in vid). A regular flow model would need to know, early on, exactly how many AAs are needed in the linker, but Branching Flows can decide on-the-fly and grow or shrink it.
You can see the branching pattern in the static plots "with trails", and if you stare at them you can see where lineages delete too.
November 10, 2025 at 9:10 AM
You can see the branching pattern in the static plots "with trails", and if you stare at them you can see where lineages delete too.
The process can be seen clearly with a QM9-trained model (continuous atom positions, discrete atom types), starting from a single atom.
November 10, 2025 at 9:10 AM
The process can be seen clearly with a QM9-trained model (continuous atom positions, discrete atom types), starting from a single atom.
Then on the internal nodes of the trees, we place "anchor" states (same space as the X1 elements). We put all that in Z.
Then, given this Z, Xt evolves over the trees, sampling when (but not which) branching and deletion events occur, all constructed to terminate at X1.
Then, given this Z, Xt evolves over the trees, sampling when (but not which) branching and deletion events occur, all constructed to terminate at X1.
November 10, 2025 at 9:10 AM
Then on the internal nodes of the trees, we place "anchor" states (same space as the X1 elements). We put all that in Z.
Then, given this Z, Xt evolves over the trees, sampling when (but not which) branching and deletion events occur, all constructed to terminate at X1.
Then, given this Z, Xt evolves over the trees, sampling when (but not which) branching and deletion events occur, all constructed to terminate at X1.
How does this work? First, you need to understand the amazing Generator Matching (arxiv.org/abs/2410.20587). In GM you first sample Z, and then construct a stochastic process Xt that, conditioned on Z, terminates at the data distribution.
November 10, 2025 at 9:10 AM
How does this work? First, you need to understand the amazing Generator Matching (arxiv.org/abs/2410.20587). In GM you first sample Z, and then construct a stochastic process Xt that, conditioned on Z, terminates at the data distribution.
Another infix, this time a nanobody CDR3.
November 10, 2025 at 9:10 AM
Another infix, this time a nanobody CDR3.
And this is "infix sampling", but where you let the model figure out how many amino acids are needed to span the gaps.
November 10, 2025 at 9:10 AM
And this is "infix sampling", but where you let the model figure out how many amino acids are needed to span the gaps.
Another binder, which threads a tail down a groove.
November 10, 2025 at 9:10 AM
Another binder, which threads a tail down a groove.
For example, here is what "binder design" looks like when you start from a single amino acid.
November 10, 2025 at 9:10 AM
For example, here is what "binder design" looks like when you start from a single amino acid.
This is our "graphical abstract":
November 10, 2025 at 9:10 AM
This is our "graphical abstract":
And here is a small molecule example.
November 10, 2025 at 9:10 AM
And here is a small molecule example.
We figured out flow matching over states that change dimension. With "Branching Flows", the model decides how big things must be! This works wherever flow matching works, with discrete, continuous, and manifold states. We think this will unlock some genuinely new capabilities.
November 10, 2025 at 9:10 AM
We figured out flow matching over states that change dimension. With "Branching Flows", the model decides how big things must be! This works wherever flow matching works, with discrete, continuous, and manifold states. We think this will unlock some genuinely new capabilities.
We will have a preprint up about this soon. This is just a "base model" so you can't really ask it to do anything specific (but stay tuned). We've put the code and weights up, so give it a spin:
github.com/MurrellGroup...
If you have a GPU, it is pretty fast.
github.com/MurrellGroup...
If you have a GPU, it is pretty fast.
June 13, 2025 at 5:06 PM
We will have a preprint up about this soon. This is just a "base model" so you can't really ask it to do anything specific (but stay tuned). We've put the code and weights up, so give it a spin:
github.com/MurrellGroup...
If you have a GPU, it is pretty fast.
github.com/MurrellGroup...
If you have a GPU, it is pretty fast.
We can mask that head's ability to attend to non-self AAs, and this prevents the model from generating symmetric backbones, without damaging the in silico refoldability of the structures themselves.
June 13, 2025 at 5:06 PM
We can mask that head's ability to attend to non-self AAs, and this prevents the model from generating symmetric backbones, without damaging the in silico refoldability of the structures themselves.
We dug in a bit, and it turns out that, among all of the layers in this network, there is a single (!!) attention head that attends to the matching residue of any symmetric copies of a chain (and also to within-chain structural repeats).
June 13, 2025 at 5:06 PM
We dug in a bit, and it turns out that, among all of the layers in this network, there is a single (!!) attention head that attends to the matching residue of any symmetric copies of a chain (and also to within-chain structural repeats).
RFdiffusion, Chroma, and others also generate symmetric structures. But, as far as we know, they only do so if some sort of symmetry constraint is imposed during generation. We aren't doing anything like that. This model just does it spontaneously!
June 13, 2025 at 5:06 PM
RFdiffusion, Chroma, and others also generate symmetric structures. But, as far as we know, they only do so if some sort of symmetry constraint is imposed during generation. We aren't doing anything like that. This model just does it spontaneously!
After checking that we weren't just mistakenly looking at the training PDBs, we realized: these are, at least approximately, exhibiting symmetry. We saw dimers, dimers of heterodimers, trimers (though the model isn't as good at these) etc.
June 13, 2025 at 5:06 PM
After checking that we weren't just mistakenly looking at the training PDBs, we realized: these are, at least approximately, exhibiting symmetry. We saw dimers, dimers of heterodimers, trimers (though the model isn't as good at these) etc.
Like MultiFlow this does struct+seq, with struct similar to FoldFlow-SFM, and with seq per Meta's Discrete Flow Matching. We trained this for a week on one RTX6000 Ada, expecting sketchy helices, and maybe the odd beta sheet pairing if we were lucky. Instead, we got these:
June 13, 2025 at 5:06 PM
Like MultiFlow this does struct+seq, with struct similar to FoldFlow-SFM, and with seq per Meta's Discrete Flow Matching. We trained this for a week on one RTX6000 Ada, expecting sketchy helices, and maybe the odd beta sheet pairing if we were lucky. Instead, we got these:
With the ecosystem all in place, this is what it takes to 1) specify the entire model, and 2) the flow matching training objective. Just mixing and matching components:
June 13, 2025 at 5:06 PM
With the ecosystem all in place, this is what it takes to 1) specify the entire model, and 2) the flow matching training objective. Just mixing and matching components:
We tried to set up a simple demo/tutorial model for the protein design ecosystem we've been developing, and it turned out a bit more interesting than we expected. 🧵
This was a team effort from a few people in my lab, including @antonoresten.bsky.social and others (not sure who is on this app)
This was a team effort from a few people in my lab, including @antonoresten.bsky.social and others (not sure who is on this app)
June 13, 2025 at 5:06 PM
We tried to set up a simple demo/tutorial model for the protein design ecosystem we've been developing, and it turned out a bit more interesting than we expected. 🧵
This was a team effort from a few people in my lab, including @antonoresten.bsky.social and others (not sure who is on this app)
This was a team effort from a few people in my lab, including @antonoresten.bsky.social and others (not sure who is on this app)
Karolinska has a rich work environment, and you'll get to collaborate closely with experimentalists. If you already have a strong biology background then you'll fit right in, and if you don't then this is an excellent way to broaden your education.
May 15, 2025 at 1:21 PM
Karolinska has a rich work environment, and you'll get to collaborate closely with experimentalists. If you already have a strong biology background then you'll fit right in, and if you don't then this is an excellent way to broaden your education.
Coming from South Africa (via five years in the US), I can say that Stockholm is a wonderful place to live. The city itself is amazing, and, if you need some nature, there are forests and water everywhere. And you never need to speak any Swedish!
May 15, 2025 at 1:21 PM
Coming from South Africa (via five years in the US), I can say that Stockholm is a wonderful place to live. The city itself is amazing, and, if you need some nature, there are forests and water everywhere. And you never need to speak any Swedish!
My lab, at Karolinska, in Stockholm, is looking for a PhD student with a computational/quantitative background to work on probabilistic/generative models of proteins (structure and sequence). The research will involve methods development, and applications in vaccine design.
May 15, 2025 at 1:21 PM
My lab, at Karolinska, in Stockholm, is looking for a PhD student with a computational/quantitative background to work on probabilistic/generative models of proteins (structure and sequence). The research will involve methods development, and applications in vaccine design.