DALTREY
All scenarios ▾

OpsPilot

Autonomous ops

SP

Payment cluster remediation

run_8f3a opspilot/remediator-v3 Needs approval
p99 latency2.84 s error rate4.1% pods ready5/6 started 14:02 · PagerDuty #4471
MC

Marcus Chenon-call SRE14:02

Payments latency is spiking, fix it

remediator-v3agent14:02

On it. p99 on payments-core is 2.84 s against a 400 ms SLO and error rate is 4.1%. Plan:

  1. Inspect pod health in prod
  2. Read recent errors from payments-core
  3. Diagnose root cause
  4. Remediate — needs approval before I touch production
tool · kubectl executed · ok · 0.4 s 
$ kubectl get pods -n prod -l app=payments
NAME                            READY   STATUS             RESTARTS   AGE
payments-core-7d9f6c4b8-2xk9p   1/1     Running            14         3d
payments-core-7d9f6c4b8-8mq2z   1/1     Running            11         3d
payments-core-7d9f6c4b8-q4nvt   0/1     CrashLoopBackOff   23         3d
payments-core-7d9f6c4b8-t7wls   1/1     Running            9          3d
payments-gateway-5b8c9d-lp3rd   1/1     Running            0          12d
payments-gateway-5b8c9d-zz8kq   1/1     Running            0          12d
tool · kubectl executed · ok · 1.1 s 
$ kubectl logs deploy/payments-core -n prod --tail=200 | grep ERROR | head -3
ERROR pool: acquire timeout after 5000ms (pool=pg-primary in_use=200/200)
ERROR vault: lease renewal failed database/creds/payments-rw (403 permission denied)
ERROR pool: acquire timeout after 5000ms (pool=pg-primary in_use=200/200)

remediator-v3agent14:03

Root cause: the Vault lease for database/creds/payments-rw failed to renew, so every pod is holding an exhausted connection pool (200/200) and one is crash-looping. Proposed remediation: rotate the credential, then rolling-restart payments-core so pods pick up the new lease.

Proposed action HIGH Privileged operation · production · irreversible 
$ vault write -f database/rotate-role/payments-rw
$ kubectl rollout restart deploy/payments-core -n prod
Target namespace
Replicas
6 pods
Blast radius
Customer-facing
Secret rotation
payments-rw revoked
Rollback
Not possible

PAY-HIL-04: the agent may prepare this change but cannot execute it. A live human must approve.