Skip to content
upcertain

Blog

Heartbeats for systemd timers, Kubernetes CronJobs and queue workers

The crontab one-liner is easy. Scheduled work increasingly runs elsewhere: systemd timers, Kubernetes CronJobs, long-running workers. Here is where the heartbeat goes in each.

Updated 25 October 2026 · By the Upcertain team

systemd timers

Put the heartbeat in the service unit, not the timer. For a Type=oneshot service, ExecStartPost= runs only after ExecStart= has finished successfully, which is exactly when you want to report in:

# /etc/systemd/system/backup.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
ExecStartPost=/usr/bin/curl -fsS -m 10 --retry 3 https://upcertain.com/api/heartbeat/<token>

Kubernetes CronJobs

A CronJob can miss its schedule without any error you would see: the controller skips runs it could not start within startingDeadlineSeconds, concurrencyPolicy: Forbid skips runs while the previous one is still going, and a suspended CronJob does nothing at all. A heartbeat at the end of the container's command catches all of these:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: export
spec:
  schedule: "15 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: export
              image: registry.example.com/export:1.4
              command: ["sh", "-c", "/app/export && wget -q -T 10 -O /dev/null \"$HEARTBEAT_URL\""]
              env:
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef: { name: upcertain, key: export-heartbeat }

Keep the URL in a Secret: anyone with it can report in for the job. Use wget or curl, whichever the image has.

Queue workers and daemons

A worker runs continuously, so there is no end of the job to report. Report from the main loop instead, about once a minute, and only while the loop is healthy:

let lastBeat = 0;
while (running) {
  const job = await queue.next({ timeoutMs: 5_000 });
  if (job) await handle(job);
  if (Date.now() - lastBeat > 60_000) {
    lastBeat = Date.now();
    fetch(process.env.HEARTBEAT_URL!, { signal: AbortSignal.timeout(5_000) }).catch(() => {});
  }
}

Report even when the queue is idle, so an idle worker is not mistaken for a dead one. Set the monitor's period to the reporting interval and a grace of one or two intervals. If the loop hangs on a stuck job, the heartbeats stop and you hear about it.

Windows Task Scheduler

& C:\jobs\import.ps1
if ($LASTEXITCODE -eq 0) {
  Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 'https://upcertain.com/api/heartbeat/<token>' | Out-Null
}

One monitor per job

Give every job its own heartbeat monitor, named after the job. An alert that says "export stopped" is actionable; one that says "something in the cluster stopped" is not. Period and grace guidance is in the dead man's switch guide, and the basics are in the heartbeat setup docs.

Questions

What if the job runs on several machines?
Give each machine's job its own heartbeat monitor if each must run, or one shared monitor if any single run is enough. A shared monitor only notices when all of them stop.
Should failures send a signal too?
Heartbeats only signal success. To hear about a failure before the deadline, have the job report errors to your error tracker as well; the heartbeat still covers jobs that never run.
What grace should a Kubernetes CronJob get?
Enough for image pulls and scheduling delays on a busy cluster, plus the job's longest normal run time. Start with the run time plus 10 minutes and tighten it after a week.
Does the heartbeat URL need authentication?
No. The token in the URL identifies the monitor, so treat the URL as a secret and keep it out of images and repositories.

Related