Manage the watcher fleet

The Watchers area is the fleet view: every registered watcher, its health, its config, deregistration, and the upload queue. Cutting releases and advertising versions is separate: Roll out watcher releases.

Monitor watcher health

Each watcher sends a heartbeat every 60 seconds with version, instrument, watch directory, upload mode, activity counters, and uptime. The fleet list surfaces:

• Status: Online, Waiting: folder not found (running, but its watch folder is missing, or went missing while it ran), Unresponsive (no heartbeat for more than 5 minutes, or more than 15 for a watcher that stopped for a Windows restart, a shutdown or an automatic update), Stopped (shut down cleanly. A stop you chose stays Stopped until the watcher starts again, and a stop for a Windows restart, a shutdown or an automatic update turns Unresponsive after 15 minutes), or Registered (no first heartbeat yet). See Watcher status.
• Change pending or Change rejected: the latest config change from the web app hasn’t reached the watcher yet, or the watcher refused it. See Change a watcher’s config.
• Edited on PC, restart pending: someone changed config.yaml on the PC while the watcher was running, and the watcher hasn’t loaded it yet. See Edits made on the PC.
• Last heartbeat: a stale value is the first sign of a stopped, disconnected, or crashed watcher
• Hostname and instrument assignment
• Events: per-watcher log (watcher_started, run_reported, file_uploaded, update_*, and error with a details.kind discriminator). Data Hub adds upload_requests_cancelled, shown as Uploads Cancelled, when someone deletes a run that has requested uploads. Expand an event to read its details. From watcher 1.3.0, a file_uploaded event reports size_bytes, part_count, prep_seconds (hashing the file and requesting its upload link), and upload_seconds (sending the file to storage). Divide size_bytes by upload_seconds to get the upload speed.

datahub.example.com/watchers

Events are the primary remote-diagnosis tool. For operator-side symptoms, see Troubleshoot a watcher.

Open a watcher to see its event log, a status chart with a connectivity strip of recent heartbeats, and its configuration.

datahub.example.com/watchers/3f2a9c1e-7b4d-4e0a-9c55-1d2e8f6a7b90

Change a watcher’s config

Admins can change a watcher’s settings from the web app, without visiting the instrument PC. This needs watcher 1.4.0 or later. For older watchers the form is turned off with a note to update the watcher.

Open the watcher and choose the Configuration tab, or choose Edit watcher config from the three-dot menu on the instrument page. You can change:

• Watch folder: any folder on the instrument PC
• File patterns
• Run detection: a preset or your own pattern, and whether to include subfolders
• Upload mode, Parallel uploads, Stability period, and Max stability wait

The form checks each value as you type. Preview with recent files shows which of the instrument’s recent files the new patterns match and the run ID each would get. It only covers files Data Hub already knows about, so files on the PC that never matched a pattern don’t appear.

After you save:

1. The change shows as Change pending.
2. Within about a minute, the watcher’s next heartbeat tells it a change is waiting. The watcher fetches it.
3. The watcher checks the change, including that the new folder exists and can be read. If a check fails, it keeps its current settings and the change shows as Change rejected with the reason, both on the Configuration tab and as an error event.
4. Otherwise it saves the new settings to config.yaml, waits up to two minutes for the uploads it’s sending, and restarts its watching with the new settings. The change shows as Applied. A file that hadn’t finished uploading, such as a large file partway through, uploads again from the start after the change, unless the change moved the watch folder or switched to manual uploads.

If the new settings match more files, the watcher reports files that are already on the PC as detected, but doesn’t upload them. To upload them, use Upload on the run page; see Request uploads. New files after the change follow the upload mode.

If the watcher is offline, the change waits and is applied when the watcher next starts. If someone also edited config.yaml on the PC while it was offline, the newer of the two edits wins and the other stays in the history.

Edits made on the PC

An edit made on the PC while the watcher is running, such as with data-hub-watcher config edit, is saved to config.yaml and recorded in the history at once, as Applied. The watcher reads config.yaml only when it starts, though, so it keeps running its previous settings until it restarts. Until then the Configuration tab shows Edited on the PC with what will change, and the watcher list shows Edited on PC, restart pending. With watcher 1.4.0 and later, Current settings shows what the watcher is running, from the revision it reports in its heartbeat.

History and restore

The History list on the Configuration tab records every change, whether it came from Data Hub or from the config file on the PC (such as data-hub-watcher config edit), with who made it and when. To go back to an earlier version, click Restore on it. That saves its settings as a new change.

When two admins edit the same watcher at once, the second save is refused with a message to reload, so neither overwrites the other unseen.

Members can see the current settings, any pending change, and the history, but only admins can change them.

Change a config with the API or an AI assistant

• API: POST /api/v1/watchers/:watcherId/config-revisions with base_revision and the full settings. Signed-in callers must be admins; tokens must list watchers:admin explicitly (legacy * and watchers:write tokens are refused). GET /api/v1/watchers/:watcherId/config-revisions lists the history. See the generated API docs.
• MCP: the update_watcher_config tool, for admins. It changes only the settings you name. The assistant shows you the change and asks you to confirm before it saves.

Deregister a watcher

Each instrument can have at most one active watcher. Deregister the old one when a PC is reimaged or replaced, or to clear “instrument already has an active watcher” on a new install.

• Web app: open the watcher under Watchers and click Deregister, or choose Deregister from the watcher’s row menu
• API: DELETE /api/v1/watchers/:watcherId (requires watchers:admin)

Deregistration is a soft-delete: heartbeats, events, and runs stay under Watchers → Deregistered. A watcher removed through the API with a token shows that token’s name as the one that removed it.

Retiring an instrument also deregisters its watchers automatically; see Retire an instrument.

You don’t need to deregister a watcher just because its token was revoked. Re-running data-hub-watcher init with a new token takes over the existing registration; see After revoking a token.

Request uploads

Upload on a run page, or on selected runs in a run list, asks the watcher to send files that are still on the instrument PC. The watcher polls the server’s upload queue once a minute and uploads the files that were requested.

• In manual upload mode, every file waits in this queue until someone requests it.
• In auto upload mode, files upload on their own, and the queue covers files that need another try. This works with watcher 1.2.1 and later; older watchers in auto mode never pick up requests, so the file shows Uploading and stays that way.

When a watcher stops cleanly and shows as Stopped, Data Hub cancels its pending requests and the files go back to detected, because nothing is left to upload them. Two stops are the exception, because the watcher is expected back: restarting the instrument PC or a full shutdown (watcher 1.4.1 and later), and a watcher that stops to install an automatic update (watcher 1.4.2 and later). The requests stay queued for the watcher to upload once it is running again, and Data Hub cancels them only if no heartbeat arrives for about 15 minutes. At that point the watcher shows as Unresponsive instead of Stopped. See When the PC restarts or shuts down.

Deleting a run also cancels the pending requests for that run’s files, and the files go back to detected. Data Hub records an upload_requests_cancelled event on the instrument’s watcher with the run and the number of files, and you can filter the event log by Uploads Cancelled to find it. That event covers only cancellations caused by deleting a run. The other cancellations in this section, such as a watcher stopping or being removed or a change of watch folder, still show under their own event types. Restoring the run doesn’t bring the requests back, so request the uploads again. Other runs on the instrument keep their requests.

Approving manual uploads

For instruments in manual upload mode, the watcher reports runs but waits for server approval before uploading file bytes. Approve a file with Upload, or choose Dismiss to tell the watcher not to upload it. To act on several files at once, select them in the files table and use the bar that appears.

datahub.example.com/instruments/hina-microscope/runs/20260316_101845_330

Safeguards that keep the queue healthy:

• Changing an instrument’s watch folder, from the web app or on the PC, reverts pending upload requests to detected so stale paths drain instead of erroring
• A queued file that keeps failing (missing on disk or failing to upload) is retried on at most three polls, then cancelled server-side