SeamlessGen: Steering LLM Generation Midstream

I built SeamlessGen to explore a simple question: what if we could change the instructions that matter while an LLM is still generating its answer?

Usually, we put all the instructions in a prompt and let the model generate. But different parts of the answer can have different requirements. When generating a webpage, the HTML structure, CSS and JavaScript do not necessarily need the same instructions, temperature or even model.

SeamlessGen uses stop sequences and assistant prefill to split that generation into phases. Pause at a known boundary, adjust the request, then resume using the text generated so far. I first shared this work in February, and used Claude Code with Opus 4.5 to build the library.

Why change the prompt halfway through?

A common piece of prompting advice is to put the most important instruction at the bottom. But what happens when there are multiple important instructions, and each matters at a different point in the output?

That was the question behind this work. Instead of asking the model to keep every instruction equally relevant throughout, the application can bring a particular instruction into focus when it is needed. For example, emphasize styling guidance inside a CSS block, then remove that temporary guidance when the block ends.

This is steering through the prompt. It does not change the model's weights or directly control its internal attention. The model still has to follow the instructions, and the application still needs to check the result.

A webpage in phases

Suppose the task is to generate a single HTML file with embedded CSS and JavaScript. The application can watch for known boundaries in the output and use them to change the request.

Open each phase below to see what changes. This is an illustration of the control flow, not a live model run.

The distinction is useful: it looks like one continuous document, but there are multiple API requests underneath. Assistant prefill supplies the unfinished response as the prefix for the next generation. It does not preserve a running computation between calls.

The repository's HTML example shows a model switch at the CSS boundary. The state-machine example in the README extends this to separate HTML, CSS and JavaScript states. Closing a style block returns to HTML; it does not by itself mean that JavaScript has started.

One mechanism, several uses

Once the application can stop and resume generation, a few other patterns become possible. Tool instructions can be introduced when the relevant tool is needed. Temporary instructions can be removed after a phase. Different sections can use different generation settings without asking for separate finished documents and combining them afterward.

The repository also includes streaming examples that detect patterns in the generated text. These make the trigger more flexible than an exact stop sequence, but the same question remains: what should change at this point, and what context should the next request receive?

Very helpful to think about, especially for smaller models dealing with several competing instructions. Whether it improves quality or cost for a particular task needs to be measured. Switching to a smaller model for one section does not automatically make the whole workflow cheaper, since the extra requests also have a cost.

A concrete experiment: repairing Mermaid diagrams

The Mermaid example looks at what happens after generated code fails validation. A usual retry includes the broken code and the parser error in the conversation. The alternative in this example is to discard the failed output, add a specific hint, and generate again from a clean request.

For example, a label containing nested square brackets can confuse Mermaid's syntax. A useful hint is to use simple descriptive text in the label. The validator still sees the error. The model receives guidance for the next attempt rather than the growing history of failed attempts.

The documented comparison reports one successful diagram out of three for error-feedback retries and three out of three for rewind plus hints, with input-plus-output token totals of 9,558 and 1,667 respectively. The example code uses three deliberately difficult prompts, up to three attempts per prompt, and claude-3-5-haiku-latest. This is a small experiment. Passing the Mermaid renderer checks that the diagram renders, not that its meaning is correct. These figures are not a general success rate for the library.

There is another distinction here: this repair example starts a fresh generation. It demonstrates the context-management idea, but does not require the same prefill mechanism as continuing an unfinished HTML document. Keeping the failed attempt out of the next prompt and pausing a generation midstream are related choices, not identical operations.

I am interested in the practical question this raises: how much of a failed attempt should become context for the next one? The comparison gives a reason to test clean retries with targeted hints. It does not establish a particular explanation about neural circuits.

Where this fits

The original implementation targets Anthropic's API and uses assistant prefill. There is an important change since that work: Anthropic's current guidance says final-assistant prefill is no longer supported starting with Claude 4.6 models. So this is a write-up of the implemented experiment, not a drop-in recipe for the latest Claude models. Adapting it requires a compatible model and API, or a different continuation strategy that needs its own testing.

There are also ordinary engineering details to handle: delimiters, partial output, retries, validation and the cost of repeated requests. A boundary like <style> is useful in this example because it is predictable. An ambiguous trigger can interrupt the wrong part of an answer.

If a single request already does the job well, there may be no reason to add this machinery. But when a task has clear phases and different instructions matter in each one, making those phases explicit is a useful thing to explore.

That is the core idea of SeamlessGen: give the application a chance to adjust the request at the point where the next instruction becomes relevant.