Prove Training Can Resume Before Choosing Interruptible GPUs

Six hours into a training run, the machine disappears. The saved weights load and the model produces an answer. Yet the restarted job begins its learning-rate schedule again and revisits data it has already consumed. The team has preserved something valuable, but it has not demonstrated that it can continue the interrupted experiment.

This hypothetical failure is a useful starting point for choosing interruptible GPU capacity. Before comparing hourly rates, ask how much valid progress survives the loss of a worker. This guide proposes a recovery test for training and fine-tuning teams. Its architecture, examples and acceptance rules are ZharfAI engineering analysis, not a customer incident report or a measured savings claim.

Decide what “resume” promises

Loading weights for inference, starting a new experiment from existing weights, and continuing an interrupted run are different operations. All can be appropriate. The problem is silently substituting one for another. Resetting the optimizer may be an intentional experiment; it should not masquerade as an unchanged continuation because the output directory kept its name.

The PyTorch saving and loading tutorial distinguishes a general training checkpoint from model state alone, identifying optimizer state and progress information as relevant additional contents. A checkpoint should therefore be treated as a recoverable state bundle, not merely a file whose suffix looks familiar.

Write the continuation promise before choosing storage options. It should identify the run, the completed training boundary, the input-data revision and the environment in which restoration is supported. State whether the aim is exact numerical continuation under controlled conditions or continuation within a justified numerical tolerance. Neither permits silently losing required training state.

The wider open-model operations guide covers fine-tuning lineage. Recovery makes that lineage concrete at a particular point in time: these artifacts belong together and describe this completed part of this run. Files with nearby timestamps are not automatically a coherent bundle.

Inventory the state that affects the next update

Use the following as a proposed inventory, not a universal serialization format. Every applicable item needs a saved representation or a tested reconstruction method. Mark irrelevant items explicitly rather than assuming the framework handles everything.

In the Transformers v5.0.0 Trainer reference, save_only_model omits optimizer, scheduler and random-generator state and does not support training resumption. The same reference warns that ignore_data_skip=True speeds restart but changes the continuation's results. These are versioned examples, not a recommendation to install that older release.

Inspect your installed version and wrappers. A method called “save model” may create an excellent inference export while omitting the continuation bundle. Verify contents and the actual restoration path. Documentation for an underlying library does not prove that your application's helper calls it with the relevant settings.

Prefer saving at a boundary where the optimizer update has completed and the next step is unambiguous. A mid-gradient-accumulation save introduces additional obligations: accumulated gradients and progress within that accumulation must be preserved or deliberately reconstructed. A global step counter cannot stand in for unfinished work.

Make data position testable

An epoch number does not identify the next sample midway through an epoch. A step count without a recoverable ordering can also be insufficient when data is shuffled. For a changing stream, “continue with the next record” may have no stable meaning unless the run defines a source revision and replay rule.

In a small test, record non-sensitive sample identifiers for several batches after the save point. Compare those identifiers with the uninterrupted reference run. Do not put private training text into public logs merely to debug recovery. The test needs evidence of ordering, not a second uncontrolled copy of the dataset.

For streaming input, document whether replay is possible and how late-arriving records are handled. If exact replay is unavailable, keep that limitation visible in the continuation decision. A new data stream plus old weights may still be useful, but it is not proof that the original run resumed unchanged.

Bind the dataset revision, preprocessing configuration and progress representation to the checkpoint identity. If a replacement worker resolves a mutable dataset name to different contents, matching the checkpoint's step counter can give false reassurance. The test should exercise the same resolution path that replacement workers will use.

Publish only completed recovery points

A worker can disappear while writing a file or between transfers of several components. “Newest directory” is therefore not a sufficient recovery policy. Our proposed design assigns each save a distinct identity and an inventory of expected components. A save under construction must not replace the last known complete recovery point.

Capture a coherent state, transfer every required component to durable storage, and verify sizes and content identities. Only after all required writes finish successfully should the system publish completion. The publication mechanism must match the storage system's guarantees; a local file rename is not automatically an atomic publication protocol for several remote objects.

This is an engineering pattern, not a claim that every framework implements it. If your selected framework already has a completion protocol, understand and test that protocol. Adding an extra marker file does not repair an unsafe underlying save. The restorer must reject incomplete candidates and explain why it selected an earlier one.

Retain at least one previous complete bundle until its successor is accepted. Storage must also survive the failure being modeled. A folder on the same disposable disk cannot protect against losing that disk. The system recovery consistency guide addresses this wider boundary; here the scope is the training process and its necessary dependencies.

Treat interruption notices as an extra opportunity

A design that begins saving only after a termination warning depends entirely on the final window. Periodic durable checkpoints should carry the recovery promise. A warning may help finish one more save, but the previous usable bundle must not depend on that warning arriving.

AWS documents two-minute notices before Spot instances are stopped or terminated, with best-effort delivery. Hibernation begins immediately rather than providing that two-minute window. These qualifications belong in the design, not in a footnote discovered after a failed run.

Google Cloud's Spot documentation separates the notice interval from shutdown: the default interval is zero, followed by a best-effort shutdown period of up to 30 seconds. It also documents an optional 120-second notice interval. Check the machine's actual configuration rather than generalizing one provider's number to every interruptible GPU.

Measure the time for the current training step, state preparation, transfer and completion publication together. Starting a save before the deadline does not establish that it finished. Test abrupt process loss and unavailable storage as well as the graceful warning path. Ordinary failures do not have to follow the provider's planned-preemption sequence.

Restore in a genuinely fresh process

Create a short run with non-sensitive data. At a known training boundary, save a checkpoint and continue uninterrupted for several more steps. Preserve the relevant state and batch identifiers from that continuation as the comparison reference. Then recover the saved boundary in a new process and execute the same continuation.

The new process must not borrow required variables, caches or local files left behind by the first process. Use the artifact retrieval path and permissions that a replacement worker will have. Otherwise the test may succeed only because a hidden dependency survived on the original machine.

Interrupt separate test runs after a complete save, during the next write, after only some components transfer, and before completion is published. Include a case without an advance warning. Specify the expected checkpoint identity and next step before each test; seeing a loss value appear again is not the acceptance criterion.

Also test an isolated copy with a required component missing and another incompatible with the approved environment. Expect a clear rejection or a policy-defined fallback to a complete bundle, not silent optimizer initialization. Manipulate only disposable test copies. Do not corrupt real training artifacts to demonstrate that corruption is possible.

Keep loading permissions unchanged. If restoration encounters a security-related loader refusal, investigate it using the model-file trust-boundary guide. Disabling a safeguard to make the recovery test green changes the security decision; it does not demonstrate successful recovery under the original conditions.

Follow a 300-step run through an incomplete save

Consider an illustrative run that reaches step 300. Its step-240 bundle is complete, but the step-280 save was interrupted. When the worker is lost, the eligible recovery point is step 240. Sixty steps must be recomputed, not twenty. These numbers are a teaching example, not a ZharfAI benchmark.

Before doing new computation, the restorer should identify bundle 240 and the expected data and environment revisions. It then restores the training state and data position. Step 241 is the next operation. Displaying 301 on a counter without the corresponding state would not establish continuity.

Suppose the step-280 weights themselves are readable, but the optimizer state is missing. The team might choose to start a new experiment from those weights. Give that experiment a new identity and explanation. The weights' usefulness does not turn them into evidence that the original trajectory was preserved.

This scenario is also a compact acceptance test: the incomplete newer candidate is rejected, the earlier complete bundle remains usable, and the wasted-work report records sixty steps. A restorer that selects only the lexically newest filename fails a meaningful test even if that file can be opened.

Separate state correctness from numerical equality

The PyTorch 2.15 reproducibility notes do not guarantee fully reproducible results across releases or platforms, or between CPU and GPU execution even with identical seeds. Define the comparison before observing results. A small numerical difference is not automatically a recovery failure, and a similar loss value is not proof of correct restoration.

Use two layers. First check state identities, data position, counters and configuration against the continuation contract. Then compare several subsequent updates in a controlled environment. Require exact equality where that is justified; otherwise use a justified, predetermined tolerance and retain the comparison evidence.

Do not widen tolerance to conceal missing state. Resetting an optimizer, deliberately changing data or restarting a learning-rate schedule is not ordinary numerical variation. A change in hardware, framework version or distributed process count needs a separate migration test. Passing on one GPU does not validate recovery of a multi-machine training job.

Price accepted progress, not repeated work

Compare total compute, storage and transfer cost against unique accepted training progress, excluding repeatedly executed work from the denominator. Progress might mean valid steps or processed examples under a fixed contract. Final quality and delivery deadlines remain separate constraints; lower computational cost alone does not decide suitability.

For a hypothetical budget, suppose ordinary capacity completes a run in ten hours. Another offer charges half the hourly rate, but saving, restarting and recomputing increase billed time to fourteen hours. Compute cost becomes the equivalent of seven ordinary-rate hours: a 30 percent reduction, not 50 percent. Storage and transfer costs still need adding.

This is arithmetic, not a prediction of any provider's prices or interruption rate. Record replacement-capacity waiting time separately. It may not incur GPU charges while still delaying delivery; Google's cited Spot guide explicitly treats capacity as finite and not always available. Money and elapsed time need distinct budgets.

Choose checkpoint frequency from measurements. Frequent saving consumes time and write capacity; infrequent saving exposes more work to repetition. Track the age of the last complete recovery point, save-completion duration and time to the first useful post-restart step. There is no single interval in this guide that is optimal for every model.

Let the recovery result change the capacity decision

Ordinary capacity can fail too. Choosing it does not remove the need for checkpointing. The distinction is whether the workload accepts interruption and its recovery cost as part of the operating plan. Name who owns that decision and who changes the plan when replacement capacity is unavailable.

Rerun continuation tests after changes to serialization, optimizer, data loading, gradient accumulation, framework or distribution strategy. Preserve a result tied to code, data, environment, checkpoint identity and interruption scenario. An old green test name does not describe a new configuration.

Monitor incomplete saves, fallback to older bundles, recomputed steps, recovery duration and the age of the last complete checkpoint. Give changes in those measures an operational response: adjust frequency, fix transfer bottlenecks, stop a run or change capacity. Before ordering a long run, demand one complete piece of evidence: the old worker is gone, the known bundle restores in a fresh environment, and the next useful step meets the declared contract.

Source notes — reviewed October 29, 2026

These five primary references support the documented product behavior. The proposed save protocol, tests, decision table and budget examples are ZharfAI analysis. A review date is not a product release date.

• PyTorch — saving and loading models: living tutorial; model and optimizer state.
• Hugging Face — Trainer v5.0.0: versioned examples of save and resume options, not a claim about the newest release.
• AWS — Spot interruption notices: current documentation; warning limitations.
• Google Cloud — Spot VMs: current documentation; notice configuration, shutdown and capacity availability.
• PyTorch 2.15 — reproducibility: page updated May 14, 2026; numerical reproducibility limits.