Scaling & production
Alquimia Platform’s master/worker design is built for horizontal scaling, but getting the most out of production requires attention to queueing, resource limits, and failure handling.
Scaling dimensions
Section titled “Scaling dimensions”| Bottleneck | Knob |
|---|---|
| HTTP request rate | Master resources (CPU/memory), ingress capacity. The master runs exactly one replica |
| Inference throughput | Worker replicas, LLM latency, tool timeouts |
| State contention | Redis connection pool, lock TTLs |
| Audit write rate | PostgreSQL connection pool and max_connections, batching |
| Knowledge retrieval | Qdrant shard count, embedding throughput |
Worker scaling
Section titled “Worker scaling”Workers are stateless and can be scaled horizontally. Because execution can be long-running, scale workers based on throughput or queue depth rather than concurrency alone. Autoscaling on Kafka consumer lag is usually more meaningful than scaling on CPU. Workers share one consumer group, so they scale only up to the partition count of alquimia.events (6 in the shipped manifests, which is also the KEDA maxReplicaCount).
Master scaling
Section titled “Master scaling”Masters handle HTTP requests, registry operations, and event publishing. They are mostly I/O-bound. The master is the single writer of the TinyDB registry, so run exactly one master replica with the Recreate deployment strategy; scale it vertically (CPU/memory) rather than horizontally. Keep master deployments separate from workers so a slow inference does not block the public API.
Failure handling
Section titled “Failure handling”- Dead-letter queue (
alquimia.events.dlq) captures events that fail signature verification or raise during processing, so you can inspect and replay them. - Task timeouts prevent a stuck agent from holding a worker indefinitely.
- Idempotent tools make retries safe when a worker crashes mid-execution.
- Circuit breakers on external tool calls protect the system from cascading failures.
Production readiness checklist
Section titled “Production readiness checklist”- Separate master (1 replica) and worker deployments with distinct service accounts.
- PostgreSQL
max_connectionssized for(DB_POOL_SIZE + DB_POOL_OVERFLOW)× (1 master + N workers). - Master registry on persistent storage or with a backed-up S3 prefix.
- Vault dynamic credentials enabled for PostgreSQL and Redis.
- OpenTelemetry metrics, traces, and logs exported to a backend.
- Alerts configured for worker lag, error rate, and secret-fetch anomalies.
- Image signatures verified at admission.
- Backup and retention policy for PostgreSQL worklog.
- Runbook for stuck tasks and dead-letter replay.