Thoughts on LLMs

LLMs enable a whole lot that was not possible before. The amount of cognitive automation now possible is quite fabulous. They're quirky - have their strengths and weaknesses.

So much is still to be done in this field to get the best out of this technology. This post covers some thoughts and observations from working with them.

1. The fast follow era

• A great advantage of this technology is how today's weak points can become training data for the next model.
• What would have been considered the "last mile" requiring manual work can be minimal because of a general reasoning entity that can set itself up for future improvement.
• Smart synthetic data generation and other techniques play a huge part here.
• Examples like letter counting and decimal comparison are useful reminders of how capabilities can change. Training and tool augmentation offer ways to address such weaknesses, though observing an improvement does not tell us which change caused it.

2. Tool augmentation

• LLMs are great routers.
• Augment them with helpful deterministic tools and you can reliably do a whole lot more.

3. Jagged intelligence

• The above example of decimal number comparison (without tools) is a case of jagged intelligence as it has been termed.
• LLMs are not great at everything - or what most people would consider simple.
• Advances in model interpretability will likely help us deal with this.
• "When can I consult an LLM for this problem?" and "Which parts of my problem can I let the LLM confidently solve?" are meaningful questions that better interpretability can help answer.
• Circuits get activated. Consider a circuit to be an expert. How can we selectively consult different experts as and only when needed?

4. Sycophancy and sensitivity to the question

• LLMs have a tendency to agree with the user's viewpoint or please them - this can be a huge problem.
• Try noticing how your chatbot answers when you ask "Which is better: Option A or Option B?" versus "Which is better: Option B or Option A?". If the answer changes with the order, that shows position sensitivity. It does not by itself show that the model is agreeing with your viewpoint.
• This is quite a problem because we often use these chatbots as experts and accept their conclusions.
• Recursive questioning and factual grounding will become important pieces to solve this.

5. Context engineering

• These language models have finite memory and it quickly becomes critical to manage it for optimal performance.
• Models are inherently lossy.
• When building systems with LLMs, we have to take this into account and channel non-determinism carefully.
• Passing the right knowledge between subagents is hard. How can you architect systems so that LLMs don't have to decide what's relevant?
• LLM performance varies with the amount of information in its context - irrespective of whether the current problem has anything to do with what's in there.
• Adherence to instructions and tendency to complete a task list drops as context keeps increasing.
• Retrieval systems are tricky to build. Using vector embeddings over arbitrary chunks is the opposite of what most people should be doing for general RAG needs.
• Playing to the strengths of LLMs becomes important. Recent solutions do a much better job at this - for example, leveraging the model's coding and filesystem knowledge to accomplish a variety of tasks.

What is needed

• Voice-native products: Voice is a fantastic medium to communicate with AI and hear back.
• Structures and abstractions to channel non-determinism.
• Selectively directing model's attention.
• Metaprompting and tools to naturally get more out of you - help you think.
• Generative UI - dynamic interfaces (ephemeral and persistent) for effective Input-Output
• Personalization that only enhances answers when appropriate and does not skew otherwise.
• Very important time to have AI Engineers actively thinking about and solving these problems.

October 2026: putting some of this into practice

The perspective above is from January, with clarifications to the training and option-order examples. Since then, I have had a chance to explore some of these questions more concretely, particularly around context and how we interact with agents.

In February, I shared SeamlessGen, a library I built to steer generation using prefill and stop sequences. The idea was to pause at a known boundary, emphasize the instructions relevant to that phase, and resume. It is one practical way of approaching the question above about selectively directing the model's attention, through the prompt rather than by changing the model itself. I have written up the mechanism and its limits here.

I also want to make the RAG point more precise. My concern is the default of embedding arbitrary chunks and treating retrieval as solved. Embeddings are useful. I had shared an introduction to binary embeddings in May 2025 because the memory and speed tradeoffs are interesting. But making vector search more efficient and deciding what information a task needs are different questions. The latter still needs attention to document structure, the query and whether the retrieved material is enough to answer it.

Voice has become a much more concrete part of my workflow too. In December, I was using Gemini in Chrome to go through bookmarks. By September, Codex voice had replaced that use for webpage summaries. The convenience is real, though so is the friction. I have used voice agents often enough to start spelling words I expect them to get wrong.

The interface around the model matters as well. I ran into two browser-control failures where the blocker was outside the tab the agent was watching: a macOS permission prompt and a native dialog. The model could not make progress from the view it had. That is a different problem from asking it to reason harder.

There is still a whole lot to do here. I am interested in systems that give the model the relevant information, useful tools and a way to check what actually happened. Better models help, and the work around them still makes a difference to the experience.