Inspired by my experience consulting for companies and the rise of inference time compute for LLMs (as well as the acceptance of users waiting while the LLM *thinks*), I suggest many teams working on RAG should rethink if you can improve your accuracy by relaxing latency.

This really resonates with me. I spent 6 years building the AI products at my previous company and felt first hand the consequences of decisions (both good and bad). I've much more opinions on how to do certain things because of it.

Try using a local UI (I'm using msty.app), and call Claude via API. Fits my bursty nature of usage, and monthly cost is much cheaper generally. You do lose their system prompt + Artifacts but I don't miss it much.

They are the victims of their own media success, they raised at ridiculous valuation and they need to show that they disrupting the software engineering market beyond something like copilot/cursor.

Do you have the link to the public repo? Java is highly requested, we just have been focusing on validating the usefulness of the current docs before spending time adding new languages.

Cartograph solves this by automatically creating and updating both written documentation and visual representations that stay synchronized with code changes.

We are very early, but there's much more to ship. We would love to get some feedback on our product.

In complex systems, we spend most of our time understanding code - but most AI tools are focused on helping you write more code. Engineering teams struggle with outdated documentation and inconsistent system diagrams. This slows down development and makes it difficult to understand complex systems.

Congrats on the move! And your first 🦋 post.

Yeah, hot mess is the right description of it 😂. Would be interested to see if there is a nicer solution.

Yeah, I've been writing rust for a few months now for parsing codebase into a graph (cartograph.app). But, it's also a conscious decision to force myself to write "idiomatic Rust", i.e. more functional.

Do you have sources for the budget claims? I have not seen any numbers comparing models so would be interested to know.

Yeah, your feed is the training data for your brain. Curate it for what you want to see more of

How does using an assert statement work in practice? Are you using something like pytest as your benchmark runner?

Very disappointed your bio doesn't say learning machine.

Good guidelines. One thing I want to add is if you are benchmarking your system's real world performance - you probably want to keep a version of the benchmark without the easy examples removed. Especially if you need to report to non-technical management how your system is performing.

The most on-pulse demo would be on the bluesky firehose data.

Models will follow the order of the schema you specify! So just putting the COT field first works well in practice. Structured outputs also ensure that the model response can be parsed as json. Use Pydantic/Zod here.