Scale workers for throughput

Alquimia Platform separates request intake from execution. The master handles HTTP traffic and publishes events; workers consume those events and run agents. Because workers are stateless, you can scale them horizontally to match demand.

Know what to scale

Bottleneck	What to adjust
High request rate	Master resources and ingress capacity (the master always runs one replica).
Long or frequent inferences	Worker replicas and LLM throughput.
Kafka lag growing	More workers or larger consumer groups.
Slow state operations	Redis connection pool and lock settings.
High audit write rate	PostgreSQL connection pool and I/O capacity.

Scaling workers is usually the right answer when inference latency is acceptable but queue depth or Kafka lag is increasing.

Scale workers

Measure current throughput using Kafka consumer lag, task completion rate, and worker CPU utilization.
Increase worker replicas in your deployment. In Kubernetes this is typically a deployment replica count or HorizontalPodAutoscaler.
Ensure the event bus has enough partitions so added workers can consume in parallel. Workers beyond the partition count of alquimia.events (6 in the shipped manifests) stay idle.
Watch for downstream pressure on PostgreSQL, Redis, and external tool APIs as throughput rises. Each worker opens its own database pool (DB_POOL_SIZE + DB_POOL_OVERFLOW, 80 by default), so check PostgreSQL max_connections before adding workers.
Tune autoscaling triggers based on consumer lag or custom inference metrics rather than CPU alone, because agent execution is often I/O-bound.

Keep masters separate

Masters should not run agent execution. Keeping master and worker deployments separate gives you:

Independent scaling for each role.
Failure isolation — a slow agent does not block the public API.
Cleaner security boundaries, because masters need broader registry access than workers.

Production scaling checklist

Worker replicas can be added without changing the master deployment.
Autoscaling uses inference-relevant metrics, not just CPU.
Task timeouts prevent a stuck agent from holding a worker indefinitely.
Idempotent tools make retries safe when workers restart.
Observability covers queue depth, worker lag, and error rate.