Scaling & production

Alquimia Platform’s master/worker design is built for horizontal scaling, but getting the most out of production requires attention to queueing, resource limits, and failure handling.

Scaling dimensions

Bottleneck	Knob
HTTP request rate	Master replicas, ingress capacity
Inference throughput	Worker replicas, LLM latency, tool timeouts
State contention	Redis connection pool, lock TTLs
Audit write rate	PostgreSQL connection pool, batching
Knowledge retrieval	Qdrant shard count, embedding throughput

Worker scaling

Workers are stateless and can be scaled horizontally. Because execution can be long-running, scale workers based on throughput or queue depth rather than concurrency alone. Autoscaling on Kafka consumer lag is usually more meaningful than scaling on CPU.

Master scaling

Masters handle HTTP requests, registry operations, and event publishing. They are mostly I/O-bound. Scale them based on request rate and registry API load. Keep master deployments separate from workers so a slow inference does not block the public API.

Failure handling

Dead-letter queues capture events that fail repeatedly so you can inspect and replay them.
Task timeouts prevent a stuck agent from holding a worker indefinitely.
Idempotent tools make retries safe when a worker crashes mid-execution.
Circuit breakers on external tool calls protect the system from cascading failures.

Production readiness checklist

Separate master and worker deployments with distinct service accounts.
Vault dynamic credentials enabled for PostgreSQL and Redis.
OpenTelemetry metrics, traces, and logs exported to a backend.
Alerts configured for worker lag, error rate, and secret-fetch anomalies.
Image signatures verified at admission.
Backup and retention policy for PostgreSQL worklog.
Runbook for stuck tasks and dead-letter replay.