Looking is not touching.
Almost all the value in operations work is in the investigation. Almost all the risk is in the action at the end. Those are separate permissions here, so an agent can be genuinely useful long before it is allowed to change anything.
Start with the alerts nobody chases.
Every team has alerts that fire, get acknowledged, and never get looked at. Give it read-only access to your deployments, health, logs and traces and it will actually look — then write up what it found, with the evidence attached.
The credentials behind that access are read-only at the source, not read-only by request. There is nothing to grant it that would let it change something by accident.
On its own that is worth having, and it costs you no risk at all to try.
- An alert fires read only
It wakes up. Until then it was doing nothing and costing nothing.
- It gathers read only
Recent deployments, health, error rates, the traces, your runbook.
- It joins the dots read only
Against what it already knows about this service and its history.
- It writes up what it found read only
With links to the evidence, and how sure it is.
- It proposes something needs approval
A specific action, with the way to undo it attached.
- It stops needs approval
And waits for a person, for as long as that takes.
Nothing changes without approval and a way back.
A restart, a flag flip, a rollback, a scale-up — each one comes to you first, with what it affects and how to reverse it. It cannot offer you the choice without a way back, however urgent things are.
Afterwards it keeps watching for a window you set. If health does not recover, it puts things back on its own and tells you — whether or not anyone is still looking at it.
Read-only is a fine place to stay.
Watch only
Investigation and write-ups on alerts nobody has time for. No risk, immediate value, and it builds up what it knows about your systems for later.
Draft it
It prepares the change — the config, the way back, what to check afterwards — and stops. A person does the last step. Most of the work, none of the authority.
Ask me first
Narrow actions, on named systems, with a way back that has to exist before you can approve. Widen slowly, and only where its track record earns it.
Plenty of teams should stay at draft-it for production indefinitely, and there is nothing wrong with that. The point is that it is your decision, per kind of action, rather than one switch you flip and hope about.
Give it your noisiest alert.
Read-only access, a written finding, no ability to act. You will know within a week whether the analysis is any good.