Security alerts and anomalies
Security alerts indicate that something in the deployment may be misconfigured, compromised, or behaving outside expected boundaries. Treat every alert as a signal to investigate, not just a threshold to silence.
Common alerts
Section titled “Common alerts”| Alert | What it means |
|---|---|
| Secret fetch spike | A role or agent is resolving secrets much faster than usual. |
| Idle role fetching secrets | A role with no task volume is still accessing secrets. |
| Master-mode inference secret fetch | A master instance is executing agent logic, which it should not do. |
| Shield block rate jump | More requests are being blocked, possibly due to an attack or classifier drift. |
| Admission control failure | A pod or image did not pass the Kyverno policy check. |
Diagnostic steps
Section titled “Diagnostic steps”-
Identify the scope from alert labels: role, agent, agentspace, runtime mode, and source.
-
Correlate with task volume metrics to see if the anomaly matches real traffic.
-
Inspect the worklog for the affected agents and sessions.
-
Check Kafka topic ACLs and event signatures to rule out unauthorized producers.
-
Review recent deployments or spec changes that might have introduced misconfiguration.
-
Verify workload identity and Vault policies are scoped correctly.
Response actions
Section titled “Response actions”- For secret anomalies, rotate the affected credentials and narrow Vault policies.
- For master-mode inference, confirm master deployments are not running worker logic.
- For shield spikes, review classifier thresholds and recent prompt-injection patterns.
- For admission failures, check image signatures, registry allow-lists, and pod security settings.
- If compromise is suspected, isolate the affected workload and revoke its credentials.