Skip to content
SearchSubscribe
Expert · Lesson 07 — Cloud training discipline — postmortem-driven SOPs
E07Expert
Expert · Lesson 07● live

Cloud training discipline

Pod lifecycle decoupling, exit-path audits, watchdog cadences. So you don’t lose a 14-hour run.

20 min read · 90 min applyprereq: Operating 02 (postmortem discipline)

Where training runs vanish

Lost training runs are the single most expensive class of incident in agentic ML work I’ve hit. A 14-hour run that vanishes is 14 hours of compute paid for, 14 hours of wall-clock waiting, and an unknown amount of operator confidence destroyed. The same incident class repeats in different forms (laptop slept, network dropped, terminal closed, atexit fired) and each time the symptom is the same. The pod is gone, the artifacts are gone, the run has to start over.

Structural cause is almost always the same. Pod lifecycle is coupled to a process on the operator’s machine. Anything that kills that process kills the pod. The cloud platform isn’t at fault. The launcher is.

The discipline below decouples pod lifecycle from launcher process. After it lands, none of the laptop-side failures can kill a run. The pod runs to completion or to a paid-for time threshold. The operator knows within 15 minutes if anything’s wrong. Artifacts get pulled by independent infrastructure.

Decoupling pod lifecycle from launcher

Single biggest move. Three properties to enforce, in order.

PropertyWhy it mattersImplementation
Auto-destroy threshold on podWorst-case bound on lifetime if everything else failsSet at pod creation, e.g., auto_destroy_after=24h. Generous but finite.
Launcher exits after creationRemoves the launcher process as a failure pointPrint pod ID + dashboard URL; exit cleanly. No wait_for_completion().
Cleanup conditional on positive completionPod destroys only when training succeeds, not on signalsTraining script writes a “done” signal; pod self-destroys on the signal. No atexit.

Auto-destroy threshold is the safety net. If the positive-completion signal never arrives (training crashed silently, network partition, anything), the pod runs to threshold and then dies. You pay for the full threshold but the pod is bounded. Not running for a week racking up charges.

Three loss patterns

Where 14-hour runs go to die. Hover any card to see the diagnosis.

№ 01

Launcher death = pod death

claim looks like“Launcher script holds the pod handle. Laptop sleeps, network drops, terminal closes — pod terminates and the run is gone.”
what’s missingPod lifecycle is coupled to a process on the operator's machine. Anything that kills that process kills the pod. The cloud is doing its job; the launcher is the failure point.
the movePod is created with explicit auto-destroy threshold (e.g., 24h, configurable per run). Launcher exits cleanly after pod is created; it does not hold the pod handle. Pod runs to completion or to the auto-destroy threshold, whichever comes first. Watchdog independent of the launcher monitors pod health and pulls artifacts on completion.
№ 02

atexit cleanup as the lifecycle

claim looks like“Cleanup logic uses Python's atexit (or equivalent) to destroy the pod when the launcher exits. "Clean exit" looks tidy but means any exit — graceful, crash, signal — destroys the pod.”
what’s missingatexit fires on conditions that aren't "job complete." SIGHUP from the laptop sleeping, ctrl-C from a typo, an unhandled exception in the launcher — all trigger pod destroy. The pod gets destroyed at the wrong times.
the moveNever use atexit for pod cleanup. Cleanup is conditional on a positive completion signal — the training script finishes successfully, writes artifacts, signals done. If that signal isn't received, the pod runs to its auto-destroy threshold (which is paid-for time, not lost work).
№ 03

Silent watchdog failure

claim looks like“Watchdog "runs." Pod actually died 6 hours ago. Watchdog never alerted because the watchdog was running on the same machine as the launcher, and that machine slept.”
what’s missingWatchdog is co-located with the failure mode it's supposed to detect. If the launcher's machine fails, the watchdog fails the same way at the same time.
the moveWatchdog runs on independent infrastructure — a separate cloud function, a phone-side cron, anything that doesn't share fate with the launcher. Watchdog pings pod health every 5-15 minutes; alerts if pod is unhealthy or unreachable. Operator knows within 15 minutes whether a 14-hour run is alive or gone.

Fix in all three is the same. Pod has its own lifecycle, cleanup is conditional on success, watchdog runs independently. Without all three, one of the failure modes leaks through and the run vanishes.

Exit-path audit

Read the launcher with this prompt. It surfaces every code path that could destroy the pod at the wrong time.

The exit-path audit
Audit my training launcher for exit-path safety.

Read the launcher script and trace every code path
that could destroy or deallocate the pod. For each:

1. What triggers it (clean exit, signal, exception,
   timeout, atexit, finally)?
2. Is the trigger correlated with "training script
   finished successfully"?
3. If not, this is a wrong-time destroy path.
   Recommend the fix.

Also check:
- Is there an explicit auto-destroy threshold on
  the pod itself?
- Is the watchdog on independent infrastructure?
- After launcher exits cleanly, can the pod still
  be reached / monitored / artifacts pulled?

Output: a list of every wrong-time destroy path with
the specific lines and the smallest fix per line.

Walks every destroy path and grades whether the trigger correlates with training success. Wrong-time destroys surface immediately.

When I ran this for the first time I found 2 wrong-time destroy paths. Most common one is an atexit handler. Second-most-common is a try / finally with destroy in the finally. Both fire at the wrong times. Both are easy to remove once you see them.

Annotated transcript

A real conversation. I lost a 14-hour run, the agent triaged structurally, identified 3 specific changes in the launcher, and surfaced the postmortem requirement at the close. Click any pinned annotation for commentary on why each turn matters.

Triaging a lost training run

Three operator turns, three agent turns. Result: 3 launcher changes scheduled + watchdog standup + a postmortem to make the fix permanent.

  1. operator

    I lost a 14-hour training run last night. Laptop slept, pod died, run is gone. What's the discipline that prevents this?

  2. agent

    Three structural moves, in order:

    1. Pod lifecycle independent of launcher. The launcher creates the pod with an explicit auto-destroy threshold and exits cleanly. The pod runs to completion or threshold, regardless of what the launcher does after.
    2. Cleanup conditional on positive completion. Pod destroys itself only when the training script signals "done — artifacts written." Not on launcher exit, not on atexit, not on signal. Positive completion only.
    3. Watchdog on independent infra. A second process — phone cron, cloud function, anything not on your laptop — pings pod health every 5-15 minutes. Alerts if pod is unhealthy or unreachable.

    The first move is non-negotiable. The other two harden against specific failure classes.

  3. operator

    What does my launcher look like today and what do I need to change?

  4. agent

    Read me the launcher and I'll diagnose. Three things I'm specifically looking for:

    1. Does the launcher hold the pod handle after creation, or exit cleanly?
    2. Does cleanup use atexit / signal handlers / try-finally with destroy in the finally?
    3. Is there a watchdog at all, and where does it run?

    With the launcher in front of me I can show you the exact 3-5 lines that need to change.

  5. operator

    [pastes launcher script, ~80 lines, uses atexit for pod destroy, holds the pod handle, has no watchdog]

  6. agent

    Three changes:

    1. Remove the atexit handler. It's the immediate cause of last night's loss — laptop sleep triggered atexit, atexit destroyed the pod. Replace with: pod has an explicit auto_destroy_after=24h flag set at creation. No atexit.

    2. Launcher exits after pod creation. After pod = create_pod(...) and confirming pod is healthy, the launcher prints the pod ID + dashboard URL and exits cleanly. It doesn't wait_for_completion() — that's the watchdog's job, not the launcher's.

    3. Add a watchdog on independent infra. Simplest version: a cron on a $5/mo VPS or a phone-side scheduled task that pings pod health every 10 minutes. Alerts (text / push) if unhealthy. Fancier later — start simple.

    With these three changes, last night's failure mode is gone. The pod runs to completion or to the 24h threshold — paid-for time, not lost work — and the watchdog tells you within 10 minutes if anything's wrong.

    One postmortem: write this up as an incident. Action items: the three changes above. Without the postmortem, the same class of failure recurs in 6 weeks because the structural fix never landed.

Training launcher checklist

Pre-flight checklist for any launcher before it runs in anger. I walk it once per launcher and re-walk after any structural change.

PRE-FLIGHT CHECKLIST FOR TRAINING LAUNCHER

[ ] Pod creation includes an explicit auto-destroy
    threshold (timedelta, generous but finite)

[ ] Auto-destroy grace period is set (e.g., 10 min
    buffer for artifact upload)

[ ] Launcher prints pod ID + dashboard URL after
    creation

[ ] Launcher exits cleanly after pod is healthy and
    confirmed running — does NOT call
    wait_for_completion() or hold the pod handle

[ ] No atexit handlers tied to pod destruction

[ ] No try/finally with destroy in finally (cleanup
    is positive-completion only)

[ ] Training script writes an explicit "done" signal
    (file write, API call, queue message) on success

[ ] Pod self-destroys on done signal OR runs to
    auto-destroy threshold — whichever is sooner

[ ] Watchdog runs on infrastructure independent of
    the launcher's machine

[ ] Watchdog pings pod health every 5-15 minutes

[ ] Watchdog alerts (text / push / email) on
    unhealthy or unreachable pod

[ ] Watchdog state persists across pings (so multi-
    failure detection isn't lost)

[ ] Artifacts are pulled by the watchdog or pod-side
    code — never by the launcher

[ ] If this launcher has lost a run before, there is
    a postmortem with action items mapped to specific
    items in this checklist

Last item is what makes the discipline compound. A postmortem whose action items map to specific checklist items lands as structural change instead of operator memory. Re-walk the checklist before the next run.

Prompt kit

Three prompts. Audit exit paths, set auto-destroy thresholds, add watchdogs. I keep mine in the project’s CLAUDE.md.

The exit-path audit
Audit my training launcher for exit-path safety.

Read the launcher script and trace every code path
that could destroy or deallocate the pod. For each:

1. What triggers it (clean exit, signal, exception,
   timeout, atexit, finally)?
2. Is the trigger correlated with "training script
   finished successfully"?
3. If not, this is a wrong-time destroy path.
   Recommend the fix.

Also check:
- Is there an explicit auto-destroy threshold on
  the pod itself?
- Is the watchdog on independent infrastructure?
- After launcher exits cleanly, can the pod still
  be reached / monitored / artifacts pulled?

Output: a list of every wrong-time destroy path with
the specific lines and the smallest fix per line.
Set an auto-destroy threshold
Update the pod creation call to include an explicit
auto-destroy threshold.

Replace:
  pod = create_pod(image=..., gpu=...)

With:
  pod = create_pod(
    image=...,
    gpu=...,
    auto_destroy_after=timedelta(hours=24),  # adjust
                                             # per run
    auto_destroy_grace_period=timedelta(minutes=10),
  )

The threshold is the worst-case bound on pod lifetime.
Set it to: training_duration_estimate * 1.5, plus
buffer for artifact upload. Don't set it to "infinite"
or omit it — that's how forgotten pods bill for days.
Add an independent-infra watchdog
Add a watchdog that runs on infrastructure independent
of my laptop / launcher.

Simplest version (pseudocode for any cron-capable
machine):

  every 10 minutes:
    ping pod health endpoint
    if pod is unhealthy or unreachable:
      send notification (text / push / email)
      log incident with timestamp
    else:
      log healthy heartbeat

Specs:
- Runs on a machine that doesn't share fate with my
  laptop — VPS, cloud function, phone scheduled task
- Alert latency: <15 minutes between failure and
  notification
- State: persists across cron runs (so we know if
  this is the first failed ping or the 6th)
- Cleanup: when pod completes successfully, watchdog
  exits cleanly and stops pinging

Apply this — harden one launcher

~90-minute exercise.Audit, refactor, watchdog, postmortem. So you don’t lose the next run.

Your first launcher hardening pass

Each step takes 15-30 minutes. Progress saves automatically.

0/5
  1. 01Read your training launcher with the exit-path audit prompt. Identify every wrong-time destroy path.Most operators have at least one — usually atexit or a try/finally with destroy in finally. Find them all before fixing.
  2. 02Add an explicit auto-destroy threshold on the pod itself. Set it generously but finite.training_duration_estimate * 1.5 plus artifact upload buffer. Don't omit; don't set to infinite.
  3. 03Refactor the launcher to exit cleanly after pod creation. Print pod ID + dashboard URL; do not wait_for_completion.If your launcher needs to do post-training work, that work moves to the pod itself or to the watchdog — not the launcher.
  4. 04Stand up a watchdog on independent infrastructure. Simplest version first; harden later.$5/mo VPS, phone cron, cloud function — any of these work. The point is independence from the launcher's fate.
  5. 05Write a postmortem on the most recent training-run loss. Action items go in the launcher template, not in your head.Without the postmortem the structural fix doesn't compound across future runs. The discipline is to write it up.
Expert tier · what's next

After this lesson