Adding AI to an existing app: six architecture patterns

Six ways to add AI to an app you already run without rewriting it: a sidecar service, an inline call behind a feature flag, retrieval over your own data, background enrichment through a queue, an agent with tools over your existing API, and a review queue with a person in the loop. Each fits one kind of job.

Why not rewrite the app?

Because rewrites fail for reasons that have nothing to do with AI. The app you have encodes years of decisions nobody remembers making, and a rewrite rediscovers them one bug at a time. The right move is to leave the app alone and add the AI part beside it, behind a typed boundary, so the worst a bad answer can do is break one feature that a flag can turn off.

The six patterns below are the shapes that boundary can take. They are ordered by how much the AI part is allowed to do, from nothing but answering to acting on your systems. Pick the least powerful one that does the job. The three questions on line 2 of the worksheet choose for you: does the app have to wait for the answer, does the answer need your own data, and does the answer need to act on your systems.

What is the sidecar service?

Pattern 1. The AI part runs as its own small service next to the app: its own process, its own deploy, its own logs. The app calls it over HTTP or drops a message on a queue and gets a result back. The app changes by one call; the sidecar can be written in whatever language the team moves fastest in and can be swapped out without touching the app.

It fits when the app is on a stack that is hard to change, when the team that owns the app is not the team building the AI part, or when you want the AI part to fail without taking the app down. It touches nothing in the app except the one call site. What to test: that the app behaves exactly as before when the sidecar is down, slow, or returns garbage, because it will do all three. Pin the runtime and the model, and change them on purpose; where we have a named sidecar deployment, its pinned versions read Details to follow.

What is the inline call behind a flag?

Pattern 2. The app calls a hosted model directly from an existing code path, behind a feature flag, with the old path as the fallback. A support reply gets a suggested draft; a form gets a classified category; a free-text field gets a cleaned version. The model call is wrapped in a typed function with a timeout, and the flag decides whether the function runs.

It fits when the task is small, the app must wait for the answer, and the answer needs no data of yours beyond what is in the request. It touches one code path and adds one dependency. What to test: the timeout and the fallback, the cost per call at your real volume, and the evaluation set on every prompt change. What goes wrong: the call is added without a timeout, the provider has a slow hour, and every request that used to take a moment now takes as long as the provider does. The flag is the fix, and the reason to have it before you need it.

What is retrieval over your own data?

Pattern 3. The model answers from your data rather than from what it was trained on. A question comes in; the app finds the handful of documents, records or chunks most relevant to it; the model reads those and answers with a citation. The chatbot on this site works this way: it answers from the pages of the site and links to the page it used.

It fits when the answer must be grounded in your content, your policies, your records, and must say where it came from. It touches your data store, which is the point and the risk: the retrieval step must respect the same permissions the app already enforces, so a user is never shown a chunk of a record they could not open. Postgres holds the embeddings well enough to start; a separate search system is a later decision. What to test: that the retrieval returns the right documents before you test what the model says about them, because a wrong answer is usually a retrieval problem wearing a model's clothes. Keep an evaluation set of questions with the document each should cite, and score retrieval and answer separately.

What is background enrichment through a queue?

Pattern 4. Something happens in the app, a record is created or updated, and a message goes on a queue. A worker picks it up, calls the model, and writes the result back to the record: a summary, a category, extracted fields, a draft. The app never waits, and the user sees the result when it lands.

It fits when the task can wait a few seconds or a few minutes, which is most tasks. It is the safest place to start, because a worker that fails retries, a worker that is slow delays nothing the user is looking at, and every result is written through the same service layer the app already uses. It touches the queue you have, or one you add, and a worker. What to test: idempotency, so a retried message does not write twice; the write path's validation, so a bad result is rejected the way a bad form is; and the backlog under load, so a burst does not become an hours-long queue. Whether the queue is one the app already has or one you add is a worksheet decision; where we have a named deployment with an enrichment worker, its queue reads Details to follow.

What is the agent with tools over your API?

Pattern 5. The model is given a small set of typed functions, tools, that call your existing API: look up the customer, find open slots, create the booking, hand off to a person. It reads what the user wants, decides which tools to call and in what order, and the tools do the work. The model never touches the database; it asks your API, and your API says yes or no the way it does for a person.

It fits when the task is a short conversation that ends in an action, and when the actions already exist as API calls with validation and authorization. It touches your API surface, which must already be safe to expose, because you are exposing it. What to test: everything the first-agent guide on this site tests, the intent set, the tool allow-list per intent, the guardrail on every reply, the test set of real transcripts, and shadow mode beside a person before any live action. What goes wrong: the tools are given more power than the task needs, and a confused model with a delete tool deletes. Read tools first and often; write tools last and once; a person in the loop for anything that costs money.

What is the review queue with a person in the loop?

Pattern 6. Any of the patterns above, with one change: the model's output goes to a person before it goes anywhere else. A drafted reply waits for approval. A classified record waits for a glance. A proposed booking waits for a click. The person's decision is recorded next to the model's proposal, and the pair becomes the next evaluation case.

It fits when the cost of a wrong output is high, when the task is new, or when the team does not yet trust the model on this data, which is every task on day one. It touches the app's UI, which needs a place to show the proposal and take the decision. What to test: that the queue is fast enough that reviewing is quicker than doing, or people will do instead of review. The review queue is how the other five patterns earn their way out of it: when the person approves the proposal unchanged often enough, for long enough, that class of case moves to automatic, and the record of approvals is the evidence. Where we have a named deployment that moved a case class to automatic, its approval rate and period read Pricing on request.

Where do these go wrong?

The pattern is chosen from a diagram instead of from the three questions, and an agent is built for a task a queue would have done. The boundary is written in the prompt instead of in code, and the first confused model finds the gap. There is no evaluation set, so every prompt change is an argument. There is no flag, so the first bad hour is a deploy at night. The retrieval step ignores permissions and shows a user a chunk of someone else's record. The model version is not pinned and the behavior changes on a Tuesday for no reason anyone can find.

Every one of those is a line on the worksheet. Bring line 1 and line 2 to a working session and we will say which pattern fits, what the boundary should be, and what the evaluation set should hold before anyone writes a prompt.