Troubleshooting

402 pro_required

What you see. An SDK call (start or signal a workflow, register a blueprint, rate_limit, a client read, a workflow schedule, an enqueue with unique_mode until_executing or until_executed) answers 402 with the code pro_required. The console has no workflow or alert pages.

Why. The deployment has no valid JoobQ Pro key, or it had none when the server started. A key issued to another deployment or organization does not count: the server ignores it and logs a line. Core is unaffected.

What to do.

1. GET /api/v1/capabilities (no sign-in needed): features.pro says whether this server has Pro on.
2. Give the server a key: connected to the portal it arrives with the licence heartbeat (./joobqui license:refresh runs one). A key set by hand in JOOBQ_PRO_LICENSE or JOOBQ_PRO_LICENSE_FILE is used only if it was issued to this deployment's organization, in practice its 14-day trial key.
3. Restart the web process if it started before it had the key: it turns its Pro features on when it starts (a first start checks in for the key first).
4. joobqui-selfhosted --worker processes get the Pro key from the web process through Redis, and only when they run with the same /data volume and environment as the web process; they look for it every minute until they have one, then every 15 minutes, with no restart.

See Licensing: Core and Pro.

402 paid_license_required (licence expired)

What you see. The console and the operations routes of /api/v1, and /metrics, answer 402 paid_license_required. Workers keep running.

Why. The deployment licence is unpaid, expired, unverifiable or the deployment is disabled in the portal. The SDK gateway is not gated by the licence.

What to do. ./joobqui license:status (prints eligible, status, paid_until); renew or re-enable it in the portal, then ./joobqui license:refresh (or wait up to 15 minutes for the next heartbeat). 402 sso_upgrade_required means the tier does not include single sign-on.

The server opens the setup page, or will not start without a licence

What you see. The log says "JoobQ is not set up yet" and the server serves only the setup page; or, with JOOBQUI_SETUP=off, it exits saying it is not connected to JoobQ or that the saved licence cannot be used.

Why. The server has no licence: it was never connected, its /data volume is new or was lost, or another server was connected at the same address and replaced it.

What to do. Start it without JOOBQUI_SETUP=off, enter the setup code from the log, choose Sign in with JoobQ and have an admin of your organization approve the code on the portal (or run ./joobqui connect --url https://...). Check that /data is a mounted volume that outlives the container.

Connecting the server does not finish

What you see. The setup page (or ./joobqui connect) says the code was denied, expired, or could not reach the portal; or the server stopped with "Connecting with JOOBQUI_ENROLLMENT_TOKEN failed".

What to do.

• Denied: an admin chose Deny. Get a new code and approve it yourself. If nobody in your organization denied it, tell your admins.
• Expired: a code lasts 10 minutes. Choose Get a new code.
• The portal could not be reached: the server, not your browser, must reach app.joobq.io over HTTPS. Check its outbound network and proxy.
• Your organization has connected as many servers as its licence allows: disable an old server on the licensing page.
• The enrollment token is not valid: it expired, was revoked, was used up, or is limited to another address. Create a new one on the licensing page; check that JOOBQUI_PUBLIC_URL is the address the token allows.

The licensing page says a licence is used by more instances than expected

More instances of one server checked in within the last hour than its tier expects, or they came from more than one network. Replicas and blue/green deploys explain it; so does a copy of the server's /data volume running somewhere else. If you do not recognise the instances, disable the server on the licensing page and connect it again: the copy stops being licensed. If you need more instances, ask us to raise the limit.

503 sdk_token_not_configured, 401 sdk_token_required

The server has no JOOBQUI_SDK_TOKEN (503), or the worker sends no token or a different one (401). Set the token on the server (or JOOBQUI_SDK_TOKEN_FILE) and the same value in the worker's environment, without a trailing newline. In production a token shorter than 32 characters stops the server.

422 on a blueprint (the worker logs a refused blueprint)

The gateway refused the blueprint and the worker does not send it again: a reference to an unknown or a later step, a duplicate node name, a query named state or steps, an after: naming an unknown step or making a cycle. 409 means a Crystal workflow already uses the name. Fix the definition and restart the worker. (A 5xx or network error is retried with the worker's registration.)

A job or step failed with LeaseExpired

The worker holding it stopped renewing its lease: it was killed, it lost the network, or the process was blocked for longer than the lease (lease_ttl_ms, 30 s by default). The task was queued again with one retry used; a job with none left went to the dead letter queue. A workflow step runs again after a crash only if it declares retries.

A step timed out although its worker is alive

The step has heartbeat_ms and its handler did not call ctx.heartbeat() within it (the worker's own lease renewals do not count), or the worker runs an SDK release that does not beat. Call ctx.heartbeat() more often than heartbeat_ms, or raise it. timeout_ms limits one attempt (24 h by default) and schedule_to_start_ms the wait for a worker.

A run ended timed_out

The blueprint's timeout_ms (the whole run's deadline) passed. It is off by default.

Jobs are not picked up

• No worker serves that queue or job type: compare the job's queue and job_type with what the worker registers (the Workers page lists each worker's queues).
• The queue is paused, or its concurrency limit is reached (Queues page).
• The queue has a throttle: at most one claim per period / limit.
• The worker cannot reach the server: from its host, curl -sS "$JOOBQ_GATEWAY_URL/api/v1/sdk/openapi.json" -o /dev/null -w '%{http_code}\n' must print 200. SDK workers retry quietly (250 ms, doubling up to 5 s).
• A delayed job (delay_ms, run_at_ms) waits for its time.

Schedules do not fire

• Job schedules are fired by the schedule runner of the web process and of any --worker process; a paused schedule skips its ticks, which are not made up on resume. A YAML entry that cannot run (bad pattern, zone or class) is listed with its problem.
• Workflow schedules fire only while a process runs the engine's scheduler: a joobq-pro workflow Worker, or the web process with JOOBQUI_WORKFLOW_SCHEDULER=true. GET /api/v1/workflow-scheduler says whether one is alive.

SDK steps reach Crystal workers (mixed engine versions)

A process on the same Redis conducts workflows with joobq-pro older than 0.4.1, which ignores a blueprint's task queue. Upgrade every process that conducts workflows to joobq-pro 0.4.1 or later.

The dead letter queue keeps growing

Look at the error class and frames on the dead-letter page: a non-retryable error, an exhausted retry policy, or LeaseExpired (workers dying). Re-drive after the fix (POST /api/v1/dlq/redrive), and set JOOBQUI_DLQ_RETENTION_DAYS if old entries should go.

Throughput is low

Run more worker processes first (each runs concurrency tasks), then more server instances. bench/BASELINE.md in the repository has measured numbers for the gateway and for Crystal workers.