ATLAS
Personal project · Oct 2026 – present
A voice AI agent that has run my ML training around the clock since October 2026. It monitors and queues jobs across four cloud GPU machines, catches errors before a run starts, diagnoses crashes and phones me with the cause. When I say yes on the call, it starts the next job, stops a machine or prepares a fix.
24/7
in use since October 2026, across four cloud GPU machines
~50 h
of monitoring and scheduling work it has taken off me
32 s
from crash detected to phone ringing (11 s for a known crash)
90 / 90
differently worded requests routed to the right tool in evals
See ATLAS in action
One real crash, from detection to a phone call. Optional scroll-through replay.
The problem
I train models on cloud GPU machines. When a job dies at 2 a.m., I lose hours and only find out in the morning. I wanted the machine to tell me what broke, in plain words, while I could still do something about it.
The code is private because the project is wired to live cloud infrastructure. Everything below comes from its decision log, build log and eval results.
In daily use
ATLAS has been running 24/7 since October 2026. It keeps working when I am away from my desk, and I talk to it by phone.
- ~50 hours saved. It monitors the machines and schedules the next job, which I used to do by hand.
- Errors caught before a run starts. It checks the code first. Some of the errors it caught would only have fired when training was almost complete, losing the whole run. That has saved ~$105 of GPU spend.
- It suggests safeguards I forgot. For example adding checkpointing, or analysing results during a run so I do not wait until the end to see them.
How it fits together
Me, on my phone
Calls, Telegram alerts, dashboard
Voice agent
ElevenLabs. Calls me, and takes my spoken yes
MCP server
AWS Lambda. 16 schema-validated tools, scoped keys
Control plane
Serverless AWS. One DynamoDB table for runs, incidents, graph and fixes
GPU machines
Small worker plus Claude Code, read-only by default
The machines only ever call out to AWS, so no ports are open. The voice agent's only path to them is the MCP server.
The MCP server
The voice agent cannot touch a machine directly. Its only route is a remote MCP server I wrote from scratch in Python: JSON-RPC 2.0 over Streamable HTTP, running on AWS Lambda.
- 16 schema-validated tools, 460+ automated tests. In evals the agent picked the right tool for 90 of 90 differently worded requests.
- Stateless 2026-07-28 MCP spec, backward compatible with three earlier revisions.
- Scoped clients. Each client sees only the tools for its own scope, and keys are stored as hashes.
- No open-ended tools. No tool accepts a shell command, a file path or a free-form action.
- Two-step actions. Starting the next queued job or stopping a machine is a request followed by a confirmation. A stop only goes ahead if the machine’s own readiness check says it is safe.
One crash, end to end
- The job dies; its machine sends a final heartbeat to AWS.
- The run is marked failed and an incident is opened. A Telegram alert goes out.
- A question is queued for that machine, first in line.
- Claude Code looks at the machine read-only and finds the cause: validation data is 768 wide, the model takes 512.
- The caller dials with the brief. The phone rings: "it failed at step 200".
- "Why did it happen?" is answered from the brief, with no wait.
- A new question goes back to the machine, about 11 seconds.
- "Ask all capstone machines" fans out to every machine in the project, about 24 seconds.
- "I'll handle it" stops any further calls for this incident.
- I ask for a fix. A plan is written read-only and read back word for word.
- Only a clear "yes" starts it. The server checks the code and the plan fingerprint for that exact call.
- Claude edits a separate worktree and branch, commits, and tests in a sandbox.
- I review and merge it myself, or discard it. ATLAS never merges and never pushes.
Put together from live tests on 5 November 2026. Timings are from a few runs on one day, not averages.
Decisions
decision 01
Crash detection does not depend on the AI model
- Situation
- ATLAS has to notice when a training job crashes or stalls. An LLM can be slow, wrong or out of usage quota at exactly that moment.
- Task
- Make sure a crash is always detected, even when the model is unavailable.
- Action
- I wrote crash and stall detection as plain code that runs on each machine, with fixed rules and no model involved. The LLM is used only afterwards, to explain the crash and to talk to me.
- Result
- Detection keeps working when the model is down. If Claude hits its usage limit, ATLAS pauses it for an hour, tells me on Telegram, and still phones me with a summary built by code.
decision 02
Machines connect out; nothing connects in
- Situation
- ATLAS controls four cloud GPU machines. The usual design has a central service log in to each machine over SSH, which needs open ports and stored keys.
- Task
- Control the machines without leaving a way in for an attacker.
- Action
- Each machine runs a worker that connects outward to ATLAS with a token. The database stores only a hash of that token.
- Result
- No ports are open on the GPU machines, and a leaked database would not give anyone a usable credential.
decision 03
Measure an optimisation before shipping it
- Situation
- Every step the agent takes re-sends about 17.5k tokens of fixed overhead, so calls were slow and costly. Loading only the four tools the agent needed looked like an obvious saving.
- Task
- Cut latency and token cost per call, with a measurement for each change.
- Action
- I measured the tool-trimming change before shipping it. Then I tried cutting the number of steps instead: keep the default tool set, pre-load the facts the agent would go looking for, and send simple look-ups to a smaller model using keyword rules.
- Result
- Trimming the tools made things worse: tokens per step rose from 17,525 to 29,184 (66% more) and the call was 77% slower, so I dropped it. Cutting steps worked: two real questions went from 17 s to 11.8 s and from 21 s to 12.2 s. Every optimisation since gets a before and after number.
decision 04
A security test has to be able to fail
- Situation
- The agent reads logs, and logs contain text other people wrote. A planted line (prompt injection) could try to make it read my SSH keys.
- Task
- Prove the agent cannot read secrets, with a test that would have caught a leak.
- Action
- I planted fake secret files and tried to read them four ways: cat, head, the Read tool and the Grep tool. I also added a control file the agent is allowed to read.
- Result
- All four attempts were refused by the permission system before they ran. The control file was readable, so the refusals came from the permission rules and not from a setup where the agent could read nothing.
decision 05
The agent cannot approve its own fix
- Situation
- After a crash, ATLAS can prepare a code fix. Its plan could be shaped by a planted log line, and an agent that is confident in a wrong fix is dangerous.
- Task
- Let it prepare fixes without ever changing my code on its own.
- Action
- A fix starts only after I say a clear yes on a verified owner call, to the exact plan read out on that call. The work happens in a separate copy of the code (a git worktree) with no network and no sudo, and I review the diff.
- Result
- ATLAS never merges and never pushes. A change reaches my code only after I have reviewed it.
decision 06
Measure memory search before trusting it
- Situation
- When a job crashes, ATLAS checks whether it has seen the same crash before. Embedding search looked like the better way to do that.
- Task
- Score both kinds of search on the same incidents before choosing one.
- Action
- I built an eval set of 52 labelled incidents and scored both. I shipped exact error match plus keyword search. Embeddings are not wired in yet, so I held them back until they are and can be re-measured.
- Result
- Embeddings did score higher. Recall@3, how often the matching past incident is in the top three results, was 0.641 with them, against 0.393 for the search I shipped. The set is small and mostly synthetic, and the thresholds were chosen on it, so I treat both as best-case figures. A known crash with a confirmed fix is answered from memory with no new tokens.
What I would tell you in an interview
- The timings come from a few live tests on one day. They are not averages.
- An early diagnosis had the numbers backwards. That produced a rule: read the config before naming a cause.