Bulk download walkthrough

When to use an export

An export turns a search into files. It runs in the background, so you make one request, wait, and download. Use it for a one-time load of up to 100,000 jobs, or for a large slice you want to import in one step. If you need more than 100,000, split the search by country, state, source or date and run the exports one after another.

1. Count the jobs first

Make the same filters into a count request. If the number is above 100,000, narrow the filters or plan several exports.

GET /v1/jobs/count?country=US&work_arrangement=remote&has_pay=true

The response is {"count": ..., "snapshot_watermark": "..."}.

2. Choose the fields you need

Add fields to the export body to keep the files small. job_id is always included. The allowed names are job_id, currentness, stale, company_id, employer_name, title, description, work_arrangement, locations, job_first_seen_at, curated_at, job_updated_at, postings and company. Without fields each line is a complete job.

3. Create the export

Send POST /v1/jobs/exports with a JSON body. The body is a search body (any filter, plus fields, sort and include) without limit or cursor, and an optional max_records from 1 to 100,000 (the default is 100,000).

POST /v1/jobs/exports
Content-Type: application/json

{
  "country": "US",
  "work_arrangement": "remote",
  "has_pay": true,
  "fields": ["job_id", "title", "employer_name", "locations", "postings"],
  "max_records": 50000
}

The API answers 202 Accepted with an export object whose status is queued. Keep export_id. An account can have one export queued or running at a time; a second create call returns 409 export_in_progress.

4. Wait for it to finish

Poll GET /v1/jobs/exports/{export_id} every 10 seconds or so. status moves from queued to running to completed, failed or expired. If it fails, error says why. A failure that says the catalog snapshot changed or the server restarted means you should create the export again. GET /v1/jobs/exports lists your last 20 exports.

5. Download the files

A completed export lists files. Each entry has n (file number), records, bytes and a relative url such as /v1/jobs/exports/{export_id}/files/1. Send the same bearer key when you download. Files are gzip, with one JSON job on each line, and each holds up to 10,000 jobs.

curl --fail-with-body -sS -o jobs-1.jsonl.gz \
  "https://api.headcountlabs.com/v1/jobs/exports/$EXPORT_ID/files/1" \
  -H "Authorization: Bearer $HEADCOUNTLABS_API_KEY"

gunzip -c jobs-1.jsonl.gz | head -n 1

Files stay available for 24 hours after the export completes. After that the status becomes expired and you create a new export. Code in five languages is in the code samples.

6. Keep the copy fresh

An export is a snapshot. It records snapshot_watermark, the catalog version it read. To stay current after loading it, call GET /v1/jobs/changes?since=... with a time at or before the export’s created_at, apply the events by job_id, and save each resume_cursor. The sync guide shows how to make those writes safe to repeat.

Limits and cost

• Up to 100,000 jobs per export and one export at a time for each account.
• Each file holds up to 10,000 jobs, and files are kept for 24 hours.
• Creating, polling, listing and downloading an export do not count against the per-minute request limit.
• An export is counted when it completes: one request for every 100 jobs written, at least one, at the price of the matching list or search request. A failed export is not counted. See pricing.
• If your allowance cannot cover the export, the create call returns 429 with data_limit_exceeded or rate_limited.
• HTML descriptions (description_format=html) are not available and return 503 service_unavailable.