Deployment, Latency & Scaling
Streaming UX, queues for long jobs, and handling concurrent users without blowing up costs.
Streaming
Agent tasks often take 5-30 seconds - an eternity in UI time. Streaming transforms this from 'is it broken?' to 'I can see it working.' Stream tokens as they generate, and stream tool status updates ('Searching knowledge base...', 'Found 3 results...', 'Drafting response...').
Perceived speed matters more than actual speed. An agent that streams progress updates at 15 seconds feels faster than one that returns a complete response at 10 seconds with no feedback. Users tolerate latency when they understand what is happening.
Implement streaming at two levels: token streaming for text generation (standard SSE/WebSocket) and event streaming for tool calls and status changes. The Vercel AI SDK, LangChain, and most agent frameworks support this. Make streaming the default UX, not an add-on feature.
Async jobs
Tasks that take more than 30 seconds should not block the UI. Move them to a background queue with email or webhook notification on completion. Report generation, batch processing, multi-document analysis, and complex research tasks all belong in async queues.
Common choices: Inngest (serverless-friendly, great DX), Trigger.dev (similar, strong TypeScript support), BullMQ (self-hosted Redis queue, more control). The pattern is the same: enqueue job → return job ID immediately → process in background → notify on completion.
Show job status in the UI: queued, running (with step progress), completed (with results), failed (with error and retry option). Users who submit a 5-minute analysis job and get no feedback for 5 minutes assume it failed. Progress updates retain users through long-running tasks.
5 more sections in this article
- Rate limits
- Where the time actually goes
- Failure and retry semantics
- Scaling cost before scaling traffic
- Worked example: the ticket triage agent
This is Pro content
Get Agent Factory Pro - a one-time payment for lifetime access to full articles, complete build prompts, and everything new.
Use it
The parts of the library that put this article to work.
1 more build prompt reference this article.
Published 27 July 2026. Last reviewed 17 August 2026. We re-read this library on a schedule and date every article, so you can see for yourself how current it is.