Raphael Pisoni
@4rtemi5.bsky.social
Unsupervised multimodal representation of a learning researcher.
https://www.rpisoni.dev/
https://www.rpisoni.dev/
I wanna talk to those experts you claim to have trained! Are they in the room with us now?
July 26, 2025 at 10:48 AM
I wanna talk to those experts you claim to have trained! Are they in the room with us now?
I ran my experiments on this "Gaussian-Kernel Attention" on the GPT speedrun repo by Keller Jordan on 8xH100. How much that's worth to compare against BIG models I don't know but I found it interesting so here is the code:
github.com/4rtemi5/modd...
github.com/4rtemi5/modd...
GitHub - 4rtemi5/modded-nanogpt
Contribute to 4rtemi5/modded-nanogpt development by creating an account on GitHub.
github.com
July 23, 2025 at 8:14 PM
I ran my experiments on this "Gaussian-Kernel Attention" on the GPT speedrun repo by Keller Jordan on 8xH100. How much that's worth to compare against BIG models I don't know but I found it interesting so here is the code:
github.com/4rtemi5/modd...
github.com/4rtemi5/modd...
Wow great hint! I actually had this unread paper open in a long forgotten tab. Seems like it's finally time to read it... ;)
arxiv.org/abs/1903.05662
arxiv.org/abs/1903.05662
Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
Training activation quantized neural networks involves minimizing a piecewise constant function whose gradient vanishes almost everywhere, which is undesirable for the standard back-propagation or cha...
arxiv.org
July 8, 2025 at 7:24 AM
Wow great hint! I actually had this unread paper open in a long forgotten tab. Seems like it's finally time to read it... ;)
arxiv.org/abs/1903.05662
arxiv.org/abs/1903.05662
This could be a way to nudge a neuron with a negative activation to still get a small positive gradient, potentially avoiding dead ReLUs in a more direct way.
Would this offer more granular control over learning dynamics compared to variants like Leaky ReLU?
Would this offer more granular control over learning dynamics compared to variants like Leaky ReLU?
July 8, 2025 at 5:59 AM
This could be a way to nudge a neuron with a negative activation to still get a small positive gradient, potentially avoiding dead ReLUs in a more direct way.
Would this offer more granular control over learning dynamics compared to variants like Leaky ReLU?
Would this offer more granular control over learning dynamics compared to variants like Leaky ReLU?
With non-car stuff you mean IT-startups right?
July 3, 2025 at 6:12 PM
With non-car stuff you mean IT-startups right?
Reposted by Raphael Pisoni
There is an oak forest in central France that was planted 400 years ago by Colbert so that France would have quality hard wood by the 2000s to build ships for its navy.
This is the type of long term planning that Seldonian predictions can help improving.
This is the type of long term planning that Seldonian predictions can help improving.
June 17, 2025 at 8:17 AM
There is an oak forest in central France that was planted 400 years ago by Colbert so that France would have quality hard wood by the 2000s to build ships for its navy.
This is the type of long term planning that Seldonian predictions can help improving.
This is the type of long term planning that Seldonian predictions can help improving.
Link to the paper!
arxiv.org/abs/2503.08502
arxiv.org/abs/2503.08502
The Space Between: On Folding, Symmetries and Sampling
Recent findings suggest that consecutive layers of neural networks with the ReLU activation function \emph{fold} the input space during the learning process. While many works hint at this phenomenon, ...
arxiv.org
April 18, 2025 at 11:20 AM
Link to the paper!
arxiv.org/abs/2503.08502
arxiv.org/abs/2503.08502
Super interesting. I think i'm around 1-2 on this scale but I'm limited by the complexity of the scene. When imagining only an apple it can be super realistic but for very complex things many details get lost and i have to focus for them to appear.
April 13, 2025 at 3:39 PM
Super interesting. I think i'm around 1-2 on this scale but I'm limited by the complexity of the scene. When imagining only an apple it can be super realistic but for very complex things many details get lost and i have to focus for them to appear.