⚠️🔧⌨️🔥
@plausiblyreliable.com
They could also just not let you have a conversation long enough to cut off the beginning of the window, or some apps will silently start a new one with a summary of the last one stuffed in
December 29, 2025 at 5:50 PM
They could also just not let you have a conversation long enough to cut off the beginning of the window, or some apps will silently start a new one with a summary of the last one stuffed in
*mathematically* an LLM is a function (context -> next token probabilities)
but without caching internal intermediate values (that are only valid to cache if you are strictly _extending_ the context) the compute cost for each token would grow quadratically with the context length
but without caching internal intermediate values (that are only valid to cache if you are strictly _extending_ the context) the compute cost for each token would grow quadratically with the context length
December 29, 2025 at 6:41 AM
*mathematically* an LLM is a function (context -> next token probabilities)
but without caching internal intermediate values (that are only valid to cache if you are strictly _extending_ the context) the compute cost for each token would grow quadratically with the context length
but without caching internal intermediate values (that are only valid to cache if you are strictly _extending_ the context) the compute cost for each token would grow quadratically with the context length
its impractical (expensive) to do anything other than [-N:] because of what's called the "kv-cache" - its only relatively cheap to compute the next token in a long session if you haven't changed anything in the prefix, as it reuses computed values that are only valid if the beginning has not changed
December 29, 2025 at 6:32 AM
its impractical (expensive) to do anything other than [-N:] because of what's called the "kv-cache" - its only relatively cheap to compute the next token in a long session if you haven't changed anything in the prefix, as it reuses computed values that are only valid if the beginning has not changed
the final binaries are generally not so bad, but `target` (with all the intermediate files) is even worse than `node_modules`
December 13, 2025 at 11:42 PM
the final binaries are generally not so bad, but `target` (with all the intermediate files) is even worse than `node_modules`
cuda is done with some wsl2-specific magic passthrough device the runtime libs know how to use - do _not_ try to actually install any drivers - that will break it, but things that bundle their own cuda runtime like pytorch should just work out of the box
for gui stuff it kinda speaks wayland now
for gui stuff it kinda speaks wayland now
September 13, 2025 at 8:58 PM
cuda is done with some wsl2-specific magic passthrough device the runtime libs know how to use - do _not_ try to actually install any drivers - that will break it, but things that bundle their own cuda runtime like pytorch should just work out of the box
for gui stuff it kinda speaks wayland now
for gui stuff it kinda speaks wayland now
never found a good way to make a disk visible to both windows and wsl and perform well from both - but one with a linux fs on it should be attachable to wsl like this learn.microsoft.com/en-us/window...
Get started mounting a Linux disk in WSL 2
Learn how to set up a disk mount in WSL 2 and how to access it.
learn.microsoft.com
September 13, 2025 at 8:50 PM
never found a good way to make a disk visible to both windows and wsl and perform well from both - but one with a linux fs on it should be attachable to wsl like this learn.microsoft.com/en-us/window...
yeah wsl just sucks at this case
September 13, 2025 at 8:48 PM
yeah wsl just sucks at this case
the root fs on wsl2 should act just like a regular linux fs on a vm - because it is - but permissions _are_ pretty broken on wsl1 generally and when using the wsl2 9p mounts of windows drives
September 13, 2025 at 8:46 PM
the root fs on wsl2 should act just like a regular linux fs on a vm - because it is - but permissions _are_ pretty broken on wsl1 generally and when using the wsl2 9p mounts of windows drives
wsl2 is much better and _almost_ the same as a vm - now mostly just need to remember that the windows fs mounts are not high iops/mmap-friendly (don't try to run stuff directly off them) and it doesn't run an actual init by default (but *can* be configured to run systemd)
September 13, 2025 at 8:38 PM
wsl2 is much better and _almost_ the same as a vm - now mostly just need to remember that the windows fs mounts are not high iops/mmap-friendly (don't try to run stuff directly off them) and it doesn't run an actual init by default (but *can* be configured to run systemd)
I would just look for "post training", "supervised fine-tuning" (human created example responses), "RLHF" (human rater score tuning) - "alignment" is a lot more related to "AI Safety" stuff, sometimes it means things like getting the models to reject bad requests and sometimes it means AI doomerism
September 9, 2025 at 4:43 PM
I would just look for "post training", "supervised fine-tuning" (human created example responses), "RLHF" (human rater score tuning) - "alignment" is a lot more related to "AI Safety" stuff, sometimes it means things like getting the models to reject bad requests and sometimes it means AI doomerism
The base models (rarely released anymore) almost certainly could be, the "personality" comes from the post training - additional steps at the end with examples in the target style and a bit of tuning by human raters scoring outputs
September 9, 2025 at 1:00 PM
The base models (rarely released anymore) almost certainly could be, the "personality" comes from the post training - additional steps at the end with examples in the target style and a bit of tuning by human raters scoring outputs
For anything new now we're using modal and having it write back to our own S3
August 21, 2025 at 6:31 PM
For anything new now we're using modal and having it write back to our own S3
IME anything ob GPU, even small non-LLM models are hard to run cost effectively if you have low or difficult to predict utilization
August 21, 2025 at 6:29 PM
IME anything ob GPU, even small non-LLM models are hard to run cost effectively if you have low or difficult to predict utilization
It seems like for a post-trained model, the best that you can do right now is just ask it: arxiv.org/abs/2305.14975
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
A trustworthy real-world prediction system should produce well-calibrated confidence scores; that is, its confidence in an answer should be indicative of the likelihood that the answer is correct, ena...
arxiv.org
August 8, 2025 at 11:49 PM
It seems like for a post-trained model, the best that you can do right now is just ask it: arxiv.org/abs/2305.14975
The first place I saw this was the GPT-4 technical report. arxiv.org/pdf/2303.08774 p.12
August 8, 2025 at 11:46 PM
The first place I saw this was the GPT-4 technical report. arxiv.org/pdf/2303.08774 p.12
One interesting result I've seen is that *base* models' (pure next token predictors) outputted probabilities match up pretty well with the likelihood of correctness and can kind of be interpreted as confidence scores, but after the post-training steps, especially RL, that stops working.
August 8, 2025 at 11:41 PM
One interesting result I've seen is that *base* models' (pure next token predictors) outputted probabilities match up pretty well with the likelihood of correctness and can kind of be interpreted as confidence scores, but after the post-training steps, especially RL, that stops working.
for the most part you should just be able to take existing web/html apps into it unmodified but it also has some escape hatches to get at native stuff if you need
August 5, 2025 at 5:50 PM
for the most part you should just be able to take existing web/html apps into it unmodified but it also has some escape hatches to get at native stuff if you need
Might be looking for something like Tauri v2.tauri.app
Tauri 2.0
The cross-platform app building toolkit
v2.tauri.app
August 5, 2025 at 5:48 PM
Might be looking for something like Tauri v2.tauri.app
They do have the option of just using claude code at API rates, fully usage based - but nobody really likes that either because you have no idea how much it will spend on a task ahead of time (and if you max out the limits subs are *still* much cheaper than API rates)
July 29, 2025 at 2:49 PM
They do have the option of just using claude code at API rates, fully usage based - but nobody really likes that either because you have no idea how much it will spend on a task ahead of time (and if you max out the limits subs are *still* much cheaper than API rates)
I like the open source vibecoding tools like Cline where you bring your own API keys better but paying the raw API prices can be rough, I use Cursor basically *for* the subsidy
July 9, 2025 at 2:45 AM
I like the open source vibecoding tools like Cline where you bring your own API keys better but paying the raw API prices can be rough, I use Cursor basically *for* the subsidy
Nowadays, after training them on the whole Internet, they do a much shorter post training phase with chat transcripts (outsourced human workers write these, usually for pennies) to make them chatbots out of the box, (ChatGPT) but even those kinds are still fundamentally text completion systems
July 9, 2025 at 1:45 AM
Nowadays, after training them on the whole Internet, they do a much shorter post training phase with chat transcripts (outsourced human workers write these, usually for pennies) to make them chatbots out of the box, (ChatGPT) but even those kinds are still fundamentally text completion systems
The actual math part of an LLM is barely a screen full of code. The behavior really is all in the training data selection and prompting.
July 9, 2025 at 1:12 AM
The actual math part of an LLM is barely a screen full of code. The behavior really is all in the training data selection and prompting.
They know what it currently is they don't know what it used to be
July 9, 2025 at 1:08 AM
They know what it currently is they don't know what it used to be
All of the chatbot ones do. Before chatbots "LLM" referred to large text auto completion systems trained on the Internet (e.g. GPT2&3). Since those can reliably complete all sorts of text, it was figured out that you could make them into chat bots by just prompting them with enough chat transcript.
July 9, 2025 at 1:08 AM
All of the chatbot ones do. Before chatbots "LLM" referred to large text auto completion systems trained on the Internet (e.g. GPT2&3). Since those can reliably complete all sorts of text, it was figured out that you could make them into chat bots by just prompting them with enough chat transcript.
it is kind of interesting that the language-based instruction of LLMs may be revealing certain truths hidden in the way we’ve communicated for decades, like how “politically incorrect” is interpreted to mean “openly racist”
Annnd @xai just deleted Musk's new prompt which told the Grok LLM to "not shy away from making claims which are politically incorrect, so long as they are well substantiated."
That prompt seemingly caused the LLM to become Nazi 4Chan, and has now been deleted, but other recent changes remain.
That prompt seemingly caused the LLM to become Nazi 4Chan, and has now been deleted, but other recent changes remain.
July 9, 2025 at 12:58 AM