Troubleshooting

Start with the flight. Model, HTTP API and A2A responses carry x-ct-flight-id (MCP responses, /v1/models and token counting don't; an MCP refusal carries flight_id in its text). Search for it under Flights to see which key made the call, where it was routed, which gate decided it and what the provider answered. Server logs are on stdout (docker logs controltower); --detailed_debug or CT_LOG_LEVEL=debug adds detail.

Asking for help? Attach a support bundle. It holds the version, settings by name, database state, health and the last day's error counts, and never includes keys, prompts or names: docker exec controltower node dist/server.mjs --support-bundle > support-bundle.json. See SUPPORT.md.

#The agent gets an error

Every refusal has a code and a message meant to be read by the agent (or its model). On model calls and HTTP APIs, a key's rate-limit 429 also carries retry-after.

#Model calls

/v1/…, /gemini/… and /bedrock/…, in the error envelope of the API the client speaks.

#MCP tool calls

A refused tool call is a tool result with isError: true, so the model can read it. Its text ends with JSON: ct_status, flight_id, and details. The flight records the code in the second column.

Some problems are JSON-RPC errors instead: -32001 (HTTP 401) for a missing or unknown key, -32004 (404) for an unknown server slug, -32601 for a method that isn't there, -32602 for a tool name without server__, -32000 while restarting, -32603 (internal_error in Flights) for anything unexpected. GET /mcp answers 405: Control Tower doesn't open server-initiated streams, so use POST.

#HTTP APIs

/http/<slug>/… refusals are {"error": {code, message, ct_status, flight_id}} with the code in an x-ct-status header too.

#A2A agents

A2A refusals are JSON-RPC errors; the reason in data is the code in capitals: AGENT_NOT_FOUND, TOOL_NOT_ALLOWED, POLICY_DENIED, APPROVAL_REQUIRED (with a ticket), APPROVAL_PENDING, CONTENT_BLOCKED, RATE_LIMIT_EXCEEDED, SHUTTING_DOWN, the DELEGATION_… codes, INVALID_WEBHOOK, INVALID_AGENT_RESPONSE, UNEXPECTED_STREAM, UPSTREAM_TIMEOUT, UPSTREAM_UNREACHABLE. A stream that fails partway is recorded as stream_error, and a task the agent reports failed or rejected as agent_task_failed / agent_task_rejected. See A2A troubleshooting.

#The console and the admin API

#Setup and deployment

Streaming stops, or approvals time out, behind a proxy. Reverse proxies buffer responses and cut long requests. Turn buffering off for Control Tower and allow at least the approval hold time (20 s by default) plus model latency — for nginx:

location / {
  proxy_pass http://controltower:4000;
  proxy_buffering off;
  proxy_read_timeout 600s;
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;     # the console's live updates use a WebSocket
  proxy_set_header Connection "upgrade";
}

Everything is gone after an upgrade. /data wasn't on a named volume, so the new container started with an empty one. Always run with -v controltower-data:/data (or a platform disk mounted at /data). See Install.

"CT_MASTER_KEY differs from /data/master.key. Refusing to start". Both are set and they disagree; stored credentials are encrypted with one of them. Keep the one your data was written with and remove the other.

A console password is lost. An admin resets it under People (a new one-time password). If the only admin lost theirs, reset it with the admin key: see People and roles.

The setup page asks for a setup code. It is printed in the server's log at start, under Setup code, with an Open link that fills it in (docker logs <container>; on a platform, its deploy logs). It is there so that whoever reaches a new install first can't create its admin. See First-run setup.

The master key is lost. Stored provider, tool-server and channel credentials can't be decrypted. Start with a new key and enter those credentials again — keys, gates, zones and history are not encrypted and are kept.

The container can't write /data. The image fixes the ownership of a root-owned volume at startup. If your platform runs containers with a fixed non-root user and read-only ownership, make the volume writable by uid 1000 (node).

Providers time out behind a corporate firewall. Set HTTPS_PROXY (and NO_PROXY for internal hosts) so Control Tower's own calls go through your proxy — see Install.

Port 4000 is taken. Publish another host port (-p 8080:4000) or set --port / CT_PORT / PORT.

--config or --policy stops startup. The log says what's wrong: a file that can't be read or parsed, or a policy naming an agent, model or zone that doesn't exist. Nothing is half-applied; fix the file and restart.

#Console and map

The console can't sign in with admin and the master key. The account is created from the admin key only when the key is set — CT_ADMIN_KEY in the environment, or master_key in the config file — and the username is UI_USERNAME (default admin).

Links in Slack, email or webhooks point at localhost. Set CT_PUBLIC_URL to the address people use to reach the console (it's detected on Render, Fly.io and Railway).

The map is empty. Nothing has called the gateway yet — check the key's Connect panel, which turns green on the first request — or a filter is on (All / Active / Gateway / Outside). Fit brings every station into view.

A line is dashed. That traffic was reported by the agent's SDK or OpenTelemetry, not routed through the gateway, so gates can't stop it. Bring it inside on the station shows how to route it. See What is enforced.

The demo won't start. It refuses when your setup already uses one of its model or tool names, so demo traffic can never reach a real provider. The message lists the names.

#Alerts

A channel shows failing. The reason is under the channel on the Alerts page; Send test tries again.

• Slack / webhook: the URL was revoked or is unreachable from the server.
• Email: Invalid login — wrong username or password (Gmail and Microsoft 365 need an app password); a timeout — the port is blocked or wrong (587 with STARTTLS, 465 with TLS from the start); 554 No SMTP server — set one on the channel or CT_SMTP_URL.

An alert didn't fire. Check the rule's trigger and condition (N times within M minutes), whether it's paused, and its cooldown: after firing, a rule waits and then sends one digest.

#Still stuck

Search the issues or open one with the version (controltower --version), how you run it, and the flight id or log lines. Report security issues privately — see SECURITY.md.