Performance tuning

Stoa is fast when it's sized right and tuned for the workload. This page covers the levers you can pull, in roughly the order you should reach for them. The biggest single factor is hardware, so start there.

Hardware first

Most performance complaints turn out to be a sizing problem, not a tuning problem. Before spending time on the rest of this page, confirm your hardware matches the workload. The hardware sizing guide lists the three tiers we recommend, the realistic concurrent-user ceiling for each, and the model sizes they support.

If you're running a 30B model on entry-tier hardware, no amount of tuning will fix it. If you're past the concurrent-user ceiling for your tier, the symptoms (slow chats, occasional timeouts) won't go away until you add capacity.

Model size and quantization

The model you pick is the single biggest performance lever after hardware.

• Smaller models run dramatically faster. An 8B or 9B parameter model runs several times faster than a 32B model on the same GPU, and several times faster again than a 70B. For most chat workloads, the smaller model is also good enough.
• Quantization trades quality for speed. A model quantized to Q4 (4-bit) runs roughly twice as fast as the same model at Q8 (8-bit), and uses about half the GPU memory. The quality cost is real but small for most use cases.
• Match the model to the task. Stoa lets you assign different models to different agents. Use a fast small model for ad-hoc chat and reserve the bigger one for research or drafting where quality matters more than latency.

WARNING

In regulated environments, performance is second to correctness. Don't quantize so aggressively that citation accuracy or factual grounding starts to suffer. Do a side-by-side test on your real KBs before committing to a quantization level.

KB indexing

Two settings affect how knowledge bases are indexed and how well they retrieve.

• Chunk size. Larger chunks capture more context per hit but reduce precision. Smaller chunks improve precision but can fragment ideas. The default is sensible for general documents. For dense technical content (contracts, specifications, policies), smaller chunks usually retrieve better.
• Chunk overlap. Some overlap helps the retriever find passages that straddle a boundary. Too much inflates index size and slows ingestion.

Both are deployment-wide settings in the admin Configuration screen, not a per-knowledge-base choice. Changing them only affects documents ingested afterward, so tune for your document profile, then reindex the knowledge bases you want brought in line with the new setting.

Concurrent user planning

Each hardware tier has a realistic concurrent-user ceiling: entry handles 5 to 10, growth 25 to 35, scale 50 to 100+. These aren't hard caps, the system won't crash when you exceed them, but slow chats, timeouts, and degraded warnings on System status will show up. If peak concurrency is creeping up, plan for the next tier before users complain. The fix is hardware, not config.

GPU memory headroom

GPU memory is the resource that fails fastest under pressure. The model takes up most of it, the rest is used for active inference contexts.

• Leave 10 to 15% headroom. A user burst, a long context window, or a background research job can push you over.
• Watch at peak hours, not at idle. The idle reading is the floor, not the average.
• Consistently above 90% at peak means you're running too large a model. Step down a quantization level or pick a smaller model.

Multi-GPU pooling

Stoa supports multi-GPU systems and can pool GPUs into a single inference host. If you already own a workstation with two consumer cards, or a server with several mid-range GPUs sitting idle, your delivery engineer can configure Stoa to use them as one pool. This is the cheapest path from entry to growth tier for organizations that already have GPU hardware. A second GPU roughly doubles inference throughput on most models. Mixing GPU generations works, but the slowest card sets the pace for any model spread across them.

Background workers

Research mode runs as a background job rather than blocking the chat. The number of background workers determines how many research tasks can run in parallel without slowing each other down. Too few and jobs queue up. Too many and they compete for the same GPU, hurting both research and live chats. A reasonable starting point is one worker per available GPU at the entry and growth tiers, two per GPU at the scale tier.

Caching

Stoa caches the text it reads out of chat attachments and fetched web pages, so sharing the same document or link into a conversation more than once doesn't cost a re-read (or, for images, another model call to describe it) every time. The cache rebuilds lazily: clearing it is safe, the next request that needs a file just reads it again. You can see how much it holds and clear it yourself from the Cache storage panel on System status, no need to involve your delivery engineer for this one.

A note on correctness vs speed

It's tempting to crank every speed lever to the maximum. In a regulated environment, that's a mistake. Quantization and smaller models both trade some quality for speed, and the quality cost can show up as wrong citations or hallucinations that nobody catches until they're already in a deliverable. Set up A/B comparisons on your real KBs before changing defaults. Speed is easy to measure, quality is not, and quality is what your users are paying for.

Next

• What to monitor to know whether your tuning helped.
• Hardware sizing when tuning hits a ceiling.
• Troubleshooting for specific slow-chat scenarios.