Kenneth Harris
@kenneth-harris.bsky.social
Neuroscientist and University College London
Perhaps the kind of thinking the LLMs used is like a kind of search. Perhaps the kind humans usually use is too.
The question of what it "really is" will be very difficult to answer or even ask precisely. The question of how to most effectively model it to get results, is easier.
The question of what it "really is" will be very difficult to answer or even ask precisely. The question of how to most effectively model it to get results, is easier.
November 21, 2025 at 6:45 PM
Perhaps the kind of thinking the LLMs used is like a kind of search. Perhaps the kind humans usually use is too.
The question of what it "really is" will be very difficult to answer or even ask precisely. The question of how to most effectively model it to get results, is easier.
The question of what it "really is" will be very difficult to answer or even ask precisely. The question of how to most effectively model it to get results, is easier.
Something like the numpy/JAX thing also happens in humans.
I have a colleague who is a native French speaker, but can only think about science in English. Not because the are not enough French words, but because the thoughts don't come in French.
I have a colleague who is a native French speaker, but can only think about science in English. Not because the are not enough French words, but because the thoughts don't come in French.
November 21, 2025 at 6:45 PM
Something like the numpy/JAX thing also happens in humans.
I have a colleague who is a native French speaker, but can only think about science in English. Not because the are not enough French words, but because the thoughts don't come in French.
I have a colleague who is a native French speaker, but can only think about science in English. Not because the are not enough French words, but because the thoughts don't come in French.
I find it effective to use an intentional stance, i.e. to model them similarly to humans. I believe this gets better results than thinking of them as algorithms like search.
This is not about what they *really are*, it is about what gets good results, for this human user.
This is not about what they *really are*, it is about what gets good results, for this human user.
November 21, 2025 at 6:45 PM
I find it effective to use an intentional stance, i.e. to model them similarly to humans. I believe this gets better results than thinking of them as algorithms like search.
This is not about what they *really are*, it is about what gets good results, for this human user.
This is not about what they *really are*, it is about what gets good results, for this human user.
Yes!
The key question is what mental model should we humans have of LLMs, to allow us to get the best out of them. ("LLM psychology")
The key question is what mental model should we humans have of LLMs, to allow us to get the best out of them. ("LLM psychology")
November 21, 2025 at 6:45 PM
Yes!
The key question is what mental model should we humans have of LLMs, to allow us to get the best out of them. ("LLM psychology")
The key question is what mental model should we humans have of LLMs, to allow us to get the best out of them. ("LLM psychology")
More boldly: modern LLMs actually think. They are trained by RL to solve problems outside their training data. They can come up with proofs of new theorems. Surely they can think of an equation to fit data and code it up.
Maybe this is saying the same as you but in anthropomorphic language.
Maybe this is saying the same as you but in anthropomorphic language.
November 21, 2025 at 3:09 PM
More boldly: modern LLMs actually think. They are trained by RL to solve problems outside their training data. They can come up with proofs of new theorems. Surely they can think of an equation to fit data and code it up.
Maybe this is saying the same as you but in anthropomorphic language.
Maybe this is saying the same as you but in anthropomorphic language.
To your original point: the relevant training data is certainly more than just neuroscience, but also more than just function fitting. It's any code that implemented any equation for any purpose.
November 21, 2025 at 3:09 PM
To your original point: the relevant training data is certainly more than just neuroscience, but also more than just function fitting. It's any code that implemented any equation for any purpose.
Sometimes it feels like the LLM is a Prima Donna, and the prompt is a motivational pep talk putting it in just the right frame of mind to get peak performance.
November 21, 2025 at 3:09 PM
Sometimes it feels like the LLM is a Prima Donna, and the prompt is a motivational pep talk putting it in just the right frame of mind to get peak performance.
2. The gradient ascent ran in JAX. When we asked the LLM to write the code in JAX, it got less creative. Perhaps because JAX code in the training set is mainly pure ML, but numpy code is science.
Solution: we asked it to write in numpy, then had yet another LLM instance translate numpy to JAX!
Solution: we asked it to write in numpy, then had yet another LLM instance translate numpy to JAX!
November 21, 2025 at 3:09 PM
2. The gradient ascent ran in JAX. When we asked the LLM to write the code in JAX, it got less creative. Perhaps because JAX code in the training set is mainly pure ML, but numpy code is science.
Solution: we asked it to write in numpy, then had yet another LLM instance translate numpy to JAX!
Solution: we asked it to write in numpy, then had yet another LLM instance translate numpy to JAX!
Two annectodal observations:
1. We tried asking the LLM to do both of these things in one prompt. It didn't work as well. Perhaps when the LLM "thought" it had to also write a fitting function, it became too "conservative" to creatively come up with weird functions.
1. We tried asking the LLM to do both of these things in one prompt. It didn't work as well. Perhaps when the LLM "thought" it had to also write a fitting function, it became too "conservative" to creatively come up with weird functions.
November 21, 2025 at 3:09 PM
Two annectodal observations:
1. We tried asking the LLM to do both of these things in one prompt. It didn't work as well. Perhaps when the LLM "thought" it had to also write a fitting function, it became too "conservative" to creatively come up with weird functions.
1. We tried asking the LLM to do both of these things in one prompt. It didn't work as well. Perhaps when the LLM "thought" it had to also write a fitting function, it became too "conservative" to creatively come up with weird functions.
The second prompt quoted the prediction function just written, and asked a new LLM instance to find an approximate start point for gradient descent to avoid local maxima.
But the LLM never had to write code to do the final parameter search.
But the LLM never had to write code to do the final parameter search.
November 21, 2025 at 3:09 PM
The second prompt quoted the prediction function just written, and asked a new LLM instance to find an approximate start point for gradient descent to avoid local maxima.
But the LLM never had to write code to do the final parameter search.
But the LLM never had to write code to do the final parameter search.
We used two prompts. The first prompt asked the LLM for a function predicting a cell's response to a stimulus, given the cell's tuning parameters (no fitting code).
We externally fit each cell's parameters with gradient descent; this fitting code wasn't written by the LLM
We externally fit each cell's parameters with gradient descent; this fitting code wasn't written by the LLM
November 21, 2025 at 3:09 PM
We used two prompts. The first prompt asked the LLM for a function predicting a cell's response to a stimulus, given the cell's tuning parameters (no fitting code).
We externally fit each cell's parameters with gradient descent; this fitting code wasn't written by the LLM
We externally fit each cell's parameters with gradient descent; this fitting code wasn't written by the LLM
Perhaps because the AI was trained on all of science, it makes these connections between fields better.
I think what it did here was assemble several things that were used before, in a new combination, for a new purpose.
But that's what most science progress is anyway!
I think what it did here was assemble several things that were used before, in a new combination, for a new purpose.
But that's what most science progress is anyway!
November 19, 2025 at 9:50 PM
Perhaps because the AI was trained on all of science, it makes these connections between fields better.
I think what it did here was assemble several things that were used before, in a new combination, for a new purpose.
But that's what most science progress is anyway!
I think what it did here was assemble several things that were used before, in a new combination, for a new purpose.
But that's what most science progress is anyway!
Good question! I don't myself know of any uses of e^-(x^p) in neuroscience (including that paper) but it is used in other fields and has a wikipedia page.
en.wikipedia.org/wiki/Stretch...
en.wikipedia.org/wiki/Stretch...
Stretched exponential function - Wikipedia
en.wikipedia.org
November 19, 2025 at 9:50 PM
Good question! I don't myself know of any uses of e^-(x^p) in neuroscience (including that paper) but it is used in other fields and has a wikipedia page.
en.wikipedia.org/wiki/Stretch...
en.wikipedia.org/wiki/Stretch...
Investigating Power laws in Deep Representation Learning
Representation learning that leverages large-scale labelled datasets, is central to recent progress in machine learning. Access to task relevant labels at scale is often scarce or expensive, motivatin...
arxiv.org
November 17, 2025 at 12:55 PM
2. Yes, the LLMs explain their reasoning, and it makes sense They say what they saw in the graphical diagnostics and how it informed their choices. (Examples in the paper's appendix). Ablation tests confirm they really use the plots! This is what makes the search over equations practical.
November 16, 2025 at 3:11 PM
2. Yes, the LLMs explain their reasoning, and it makes sense They say what they saw in the graphical diagnostics and how it informed their choices. (Examples in the paper's appendix). Ablation tests confirm they really use the plots! This is what makes the search over equations practical.
Thanks Dan!
1. Combinatorial SR seems impractical because evaluating each function needs a non-convex gradient descent parameter search. We had the LLMs write functions estimating gradient search startpoints, which ablation tests showed was essential. Combinatorial SR couldn’t have done this.
1. Combinatorial SR seems impractical because evaluating each function needs a non-convex gradient descent parameter search. We had the LLMs write functions estimating gradient search startpoints, which ablation tests showed was essential. Combinatorial SR couldn’t have done this.
November 16, 2025 at 3:11 PM
Thanks Dan!
1. Combinatorial SR seems impractical because evaluating each function needs a non-convex gradient descent parameter search. We had the LLMs write functions estimating gradient search startpoints, which ablation tests showed was essential. Combinatorial SR couldn’t have done this.
1. Combinatorial SR seems impractical because evaluating each function needs a non-convex gradient descent parameter search. We had the LLMs write functions estimating gradient search startpoints, which ablation tests showed was essential. Combinatorial SR couldn’t have done this.
10. Finally, some thoughts. The AI scientist excelled at equation discovery because its success could be quantified. AI scientists can now help with problems like this in any research field. Interpreting the results for now still required humans. Next year, who knows.
November 14, 2025 at 6:07 PM
10. Finally, some thoughts. The AI scientist excelled at equation discovery because its success could be quantified. AI scientists can now help with problems like this in any research field. Interpreting the results for now still required humans. Next year, who knows.
9. Because the AI system gave us an explicit equation, we could make exactly solvable models for the computational benefits and potential circuit mechanisms of high-dimensional coding. Humans did this part too!
November 14, 2025 at 6:07 PM
9. Because the AI system gave us an explicit equation, we could make exactly solvable models for the computational benefits and potential circuit mechanisms of high-dimensional coding. Humans did this part too!
8. We proved that when p<2, the tuning curves’ sharp peaks yield power-law PCA spectra of exponent 2(p+1). Most cells had p~1, explaining the observed exponent of ~4. The same model explained high-dimensional coding in head-direction cells. Humans did this part.
November 14, 2025 at 6:07 PM
8. We proved that when p<2, the tuning curves’ sharp peaks yield power-law PCA spectra of exponent 2(p+1). Most cells had p~1, explaining the observed exponent of ~4. The same model explained high-dimensional coding in head-direction cells. Humans did this part.
7. The equation the AI scientist found is simple and, in retrospect, we humans could have come up it with ourselves. But we didn’t. The AI scientist did. The key innovation was replacing the Gaussian power of 2 with a parameter p controlling the peak shape.
November 14, 2025 at 6:07 PM
7. The equation the AI scientist found is simple and, in retrospect, we humans could have come up it with ourselves. But we didn’t. The AI scientist did. The key innovation was replacing the Gaussian power of 2 with a parameter p controlling the peak shape.
6. The AI scientist took 45 minutes and $8.25 in LLM tokens to find a new tuning equation that fits the data better, and predicts the population code’s high-dimensional structure – even though we had only tasked it to model single-cell tuning.
November 14, 2025 at 6:07 PM
6. The AI scientist took 45 minutes and $8.25 in LLM tokens to find a new tuning equation that fits the data better, and predicts the population code’s high-dimensional structure – even though we had only tasked it to model single-cell tuning.
5. The LLM was given fit-quality scores for previous equations, and graphical diagnostics comparing the fits to raw data. It explained how it used these graphics to improve the equations, in the docstrings of the code it wrote.
November 14, 2025 at 6:07 PM
5. The LLM was given fit-quality scores for previous equations, and graphical diagnostics comparing the fits to raw data. It explained how it used these graphics to improve the equations, in the docstrings of the code it wrote.
4. Resolving this conundrum needs a new model for orientation tuning. We found one with AI science. Equations modelling tuning curves were encoded as Python programs, and a reasoning LLM repeatedly created new models improving on previous attempts.
November 14, 2025 at 6:07 PM
4. Resolving this conundrum needs a new model for orientation tuning. We found one with AI science. Equations modelling tuning curves were encoded as Python programs, and a reasoning LLM repeatedly created new models improving on previous attempts.
3. Visual cortex encodes even low-dimensional stimuli like oriented gratings with high-dimensional power-law codes. But classical tuning models like the double Gaussian predict low-dimensional codes with exponential spectra.
November 14, 2025 at 6:07 PM
3. Visual cortex encodes even low-dimensional stimuli like oriented gratings with high-dimensional power-law codes. But classical tuning models like the double Gaussian predict low-dimensional codes with exponential spectra.