# Plan a safe data sync

A step-by-step plan for loading a slice of jobs and keeping it current without gaps or duplicates.

## Plan the sync before you start

A good sync loads once, then follows changes. Decide first which jobs you want, because every read costs requests. Write the filters down (for example `country=US` and `work_arrangement=remote`) and keep them fixed for the whole sync. A cursor belongs to one query, so changing a filter or the page size means starting again.

## Step 1: load a slice at a time

- Note the time. You will need it for the changes feed.
- Count the slice with `GET /v1/jobs/count` to estimate pages.
- Page through `GET /v1/jobs` with `limit=250`, passing each `next_cursor` back as `cursor`. Or run an [export](https://headcountlabs.com/docs/bulk-download/) when the slice is 100,000 jobs or fewer.
- Save each job under its `job_id`, replacing any earlier copy. Do not append.

Load by slices (one country, or one `updated_since` to `updated_until` range at a time). If a load fails, you only repeat one slice.

## Step 2: follow the changes feed

Start `GET /v1/jobs/changes` with `since` set to the time you noted in step 1, so nothing that changed during the load is missed. After each page, save the work first and then store the page’s `resume_cursor`. Pass it as `after` next time. Keep reading while `has_more` is true.

- Replace your copy with the `job` in `created` and `updated` events.
- Mark the job closed on an `expired` event. It carries no job object.
- Skip events whose `event_id` you have already applied.

## Make writes safe to repeat

Cursors and retries mean you will sometimes read the same job twice. If every write replaces the job by `job_id`, a repeated read changes nothing. Keep the `occurred_at` and `event_id` of the last event you applied as a fallback.

## When a cursor stops working

A jobs cursor in the default order keeps working while new data is published, but a long walk can miss a job added or updated during it. That is why you follow the walk with the changes feed. A relevance cursor, and any cursor from an older catalog version, can return `400`. Restart that listing from its first page. For the changes feed, restart with `since` set a little before the `occurred_at` of your last saved event.

## Test the failure cases

- A page you already saved arrives again.
- A cursor is rejected after a new catalog version.
- Your process stops halfway through a page.
- `429` with `Retry-After`: wait that many seconds.
- `503`: retry after a short pause.
- `503 service_unavailable` after a long wait, or `504 query_timeout`: the query hit the 90-second limit; narrow the slice.

Source: https://headcountlabs.com/docs/sync-strategy/