Provider Smoke Runbook

Use this runbook when model dropdowns are populated but you need to prove that the selected provider/model can execute through the harness gateway.

These checks call live providers. Keep them out of npm test because they depend on credentials, local services, rate limits, network access, and provider behavior.

What It Verifies

Codex executes through codex exec --model <model>.
OpenCode Zen executes through opencode run --model opencode/<model>.
OpenCode Go executes through opencode run --model alibaba-coding-plan/<model>.
Ollama executes through the local Ollama API.
Each provider can return valid JSON to the harness parser.

The smoke harness uses a temporary empty repository, asks for a minimal JSON-only response, and removes the fixture directory afterward.

Two modes are available:

gateway: direct model gateway JSON check. This is the default and is the fastest provider transport check.
workflow: full Aegis provider-smoke workflow with queued run state, workspace artifacts, trace metadata, and verifier output.

Check Readiness First

These checks do not generate model output:

curl http://127.0.0.1:4317/providers/health
curl http://127.0.0.1:4317/control-plane/status

Provider readiness includes CLI/API availability, catalog source, execution availability, credential status, and recent provider failures observed by workflow runs.

Register Provider Secrets

Secrets are optional for providers that already have external CLI auth, but the registry gives the harness a portable and inspectable credential path.

npm run cli -- set-secret \
  --name "OpenAI API key" \
  --kind provider-api-key \
  --provider openai \
  --env-var OPENAI_API_KEY \
  --value-file ~/.secrets/openai-api-key

npm run cli -- set-secret \
  --name "OpenCode API key" \
  --kind provider-api-key \
  --provider opencode-zen \
  --env-var OPENCODE_API_KEY \
  --value-file ~/.secrets/opencode-api-key

Use --input secret.json, --env-var, or dashboard Env Var when a provider needs a specific environment variable name. API and CLI responses never return raw secret values; they return metadata, SHA-256, and a redacted preview.

The smoke runner does not pass the full shell environment to provider CLIs. For direct gateway checks, it maps only the documented provider credential variables into the bounded provider subprocess environment: OPENAI_API_KEY for Codex, OPENCODE_API_KEY for OpenCode, and OLLAMA_API_KEY for Ollama.

Run A Single Provider

AGENT_HARNESS_ALLOW_LIVE_MODEL_SMOKE=1 npm run smoke:providers -- --provider codex

Full workflow mode:

AGENT_HARNESS_ALLOW_LIVE_MODEL_SMOKE=1 npm run smoke:providers -- --mode workflow --provider codex

AGENT_HARNESS_ALLOW_LIVE_MODEL_SMOKE=1 npm run smoke:providers -- --provider opencode-zen

AGENT_HARNESS_ALLOW_LIVE_MODEL_SMOKE=1 npm run smoke:providers -- --provider opencode-go

AGENT_HARNESS_ALLOW_LIVE_MODEL_SMOKE=1 npm run smoke:providers -- --provider ollama

Run Multiple Providers

AGENT_HARNESS_ALLOW_LIVE_MODEL_SMOKE=1 npm run smoke:providers -- --provider codex,opencode-zen,opencode-go

AGENT_HARNESS_ALLOW_LIVE_MODEL_SMOKE=1 npm run smoke:providers -- --provider all

Override Models

AGENT_HARNESS_ALLOW_LIVE_MODEL_SMOKE=1 npm run smoke:providers -- \
  --provider codex \
  --model codex=gpt-5.4-mini

AGENT_HARNESS_ALLOW_LIVE_MODEL_SMOKE=1 npm run smoke:providers -- \
  --provider opencode-go \
  --model opencode-go=alibaba-coding-plan/qwen3.5-plus

Equivalent environment overrides:

AGENT_HARNESS_ALLOW_LIVE_MODEL_SMOKE=1 \
AGENT_HARNESS_SMOKE_CODEX_MODEL=gpt-5.4-mini \
npm run smoke:providers -- --provider codex

Available override variables:

AGENT_HARNESS_SMOKE_CODEX_MODEL
AGENT_HARNESS_SMOKE_OLLAMA_MODEL
AGENT_HARNESS_SMOKE_OPENCODE_ZEN_MODEL
AGENT_HARNESS_SMOKE_OPENCODE_GO_MODEL

Expected Output

PASS codex (gpt-5.4-mini): connected
PASS opencode-zen (opencode/gpt-5-nano): connected
PASS opencode-go (alibaba-coding-plan/qwen3-coder-plus): connected

Failures should be treated as provider setup or gateway issues before testing full UI runs. Common causes are expired auth, missing CLI binaries, rate limits, Ollama not running, or a provider returning non-JSON text.

After A Provider Passes

Create a normal UI run at http://127.0.0.1:4317 using the same provider and model. If the smoke check passes but the workflow run fails, debug the workflow prompt, JSON schema expectations, and task-level failure banner rather than the provider connection.