Multi-provider LLM service with failover

Problem

The service takes user-written content and returns improved text, generated media and answers grounded in that user’s own material. Relying on one model or provider makes that fragile: a provider rate-limits or goes down, a paid media API runs over budget, and the whole request fails.

It also cannot trust its inputs or outputs. User content can carry personal data, prompt injection or harmful material, and generated media can come back unsafe.

How it works

An async FastAPI application. Heavy work runs as background jobs: the API validates and queues a request, a job processor runs the pipeline, and the client polls for the result.

Each text request is classified by what it needs (creative writing, tone preservation, speed, general use). That picks a task-aware route across several language models with settings per task, and media generation goes through a separate router that load-balances across multiple image providers.

• Retries with backoff handle rate limits on one provider; errors that retrying will not fix move the request to the next provider.
• A circuit breaker per provider stops calls to a provider that keeps failing, then lets a single test call through before trusting it again.
• Knowledge is chunked, embedded and deduplicated, and every chunk is tagged with its workspace, so retrieval never returns another workspace’s content.
• Media jobs are spread across the image providers by weight, and a daily spend cap (configurable per provider) switches traffic to cheaper inference once it is reached.
• Moderation, personal-data detection and prompt-injection checks run in parallel on input, and generated images and video are screened before they are returned.

Hard parts

• Keeping requests alive when a provider fails: retrying and falling over to another provider are separate steps, and the breaker keeps the router from repeatedly calling a provider that is already down.
• Recovering jobs left behind by a crashed worker, without two workers picking up the same job: recovery runs under a lease that only one worker can hold at a time.
• Choosing which way to fail: input that fails a safety check is rejected, while a moderation outage does not block content that was already generated.

My contribution

I build and run this service for the client under contract: the routing and failover engine, the per-provider circuit breakers, the retrieval layer with per-workspace isolation, the media-generation router with its spend cap, the safety screening on input and output, and the async job processor with crash recovery. I also added the Prometheus metrics, structured logging and the Docker/CI-CD delivery path. The product and its content requirements are the client’s; the service architecture and implementation described here are my work.

What it does

The service reports per-provider request counts, latency, token use and safety decisions as Prometheus metrics, so fallbacks and load balancing can be seen as they happen. Requests carry a correlation ID through the async pipeline for tracing. It runs in Docker and ships through CI/CD.

Limitations and trade-offs

• The code is private client work, so this page covers the engineering approach only and carries no reproducible repository or public metrics.
• Failover improves availability, not correctness: a request that falls over to a second provider gets a usable answer, but not necessarily the same answer the preferred model would have given.
• Asynchronous job processing adds a polling round trip and more moving parts (a queue, a worker, a recovery lease) than a synchronous call, which is the right trade for long media jobs but overhead for a trivial one.
• The media spend cap protects the budget by switching to cheaper inference once it is reached, which means output quality can step down near the limit rather than the request simply failing.
• Safety screening fails in two directions on purpose: unsafe input is rejected, but a moderation-service outage does not block content that was already generated, so that path trades strict enforcement for availability.

Artifacts

Built for a client under contract. The code is private, so this page describes the engineering approach only.