Skip to content
Labelty Docs

Monitor and build

After you start a run, monitor it until completion, then build the output model.

Poll the run

Section titled “Poll the run”

Endpoint: GET /projects/{projectId}/training/runs/{runId}

Poll every 2–5 seconds while status is QUEUED or RUNNING.

Terminal window
curl -s "$BASE/projects/$PROJECT_ID/training/runs/$RUN_ID" \
  -H "Authorization: Bearer $KEY"

What happens on each poll:

  1. Backend refreshes SageMaker job phase (starting → downloading → training → uploading).
  2. If the training worker can reach Labelty, epoch metrics merge into live_progress.
  3. metrics_history accumulates per-epoch points for charts.

Screenshot: Training monitor — epoch progress, loss cards, live log

live_progress (setup phases)

Section titled “live_progress (setup phases)”
sagemaker_phase Label Typical meaning
starting Starting Preparing instances
downloading Downloading Pulling the training image
training Training Job is training
uploading Uploading Uploading model artifacts

Show a setup stepper until training epochs appear (epoch present).

Epoch metrics (while training)

Section titled “Epoch metrics (while training)”

Common fields on live_progress / metrics_history:

  • Losses: box_loss, cls_loss, dfl_loss (and seg_loss for segmentation)
  • Validation: map50, precision, recall
  • learning_rate, gpu_mem_gb
  • Early stop: patience_current / patience_max
  • log_lines for a live log panel

After SUCCEEDED, metrics holds the final summary; training_report may include confusion matrix, per-class AP, F1 curve, and validation samples with presigned images.

Wallet stop and resume

Section titled “Wallet stop and resume”

If the prepaid wallet runs out mid-job, status may become STOPPED_EARLY with a saved checkpoint. Top up the wallet, then use the product’s resume action (API resume endpoint in the training API) to continue.

Build the inference package

Section titled “Build the inference package”

Show Build when:

  • status === "SUCCEEDED"
  • output_inference_built === false

Endpoint: POST /projects/{projectId}/training/runs/{runId}/build

Terminal window
curl -sX POST "$BASE/projects/$PROJECT_ID/training/runs/$RUN_ID/build" \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{}'

What build does: packages inference for the output model, wires label_ids, and activates the model for auto-label / deployment consumers.

Idempotent: calling build again returns success with already_built: true.

After a successful build

Section titled “After a successful build”
  1. Confirm the model appears in GET …/models?is_active=true.
  2. Run auto-label in the project UI on new assets, or
  3. Point an inference workflow deploy_config.models entry at this model UUID.

Failure checklist

Section titled “Failure checklist”
Symptom What to check
Stuck in starting / downloading SageMaker capacity / infra; wait or contact ops
FAILED with memory language Lower batch / imgsz, or pick a larger GPU tier
Empty epoch metrics but phases move Worker cannot POST progress; phases still update from SageMaker
Build missing in UI Ensure SUCCEEDED and output_inference_built is false

Next steps

Section titled “Next steps”