> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.adid.dev/llms.txt.
> For full documentation content, see https://docs.adid.dev/llms-full.txt.

# Observability

The API server uses **Uber Zap** (`go.uber.org/zap`) for structured JSON logging. Verified at `cmd/server/main.go:27` (`logger, _ := zap.NewProduction()`).

#### 14.8.1 Log levels

| Level   | When to use                                                             |
| ------- | ----------------------------------------------------------------------- |
| `debug` | Trace-level — request bodies, full SQL. Dev only.                       |
| `info`  | Default — request completed, service ready, on-chain tx receipt.        |
| `warn`  | Recoverable issues — chain client unavailable, OAuth provider degraded. |
| `error` | Bug or environment failure — handler panic, DB pool exhausted.          |
| `fatal` | Boot-time only — config load failed, DB ping failed. Process exits.     |

Set with `LOG_LEVEL=debug|info|warn|error`.

#### 14.8.2 Health probes

Two HTTP endpoints (handlers in `internal/handler/health.go`):

| Endpoint      | Used by   | Behaviour                                                                             |
| ------------- | --------- | ------------------------------------------------------------------------------------- |
| `GET /health` | Liveness  | Always returns `200 { "status": "ok" }` if the process is up.                         |
| `GET /ready`  | Readiness | Pings Postgres + Redis; returns `200` only when both are up. Returns `503` otherwise. |

Map these to Kubernetes:

```yaml
livenessProbe:
  httpGet: { path: /health, port: 8080 }
readinessProbe:
  httpGet: { path: /ready,  port: 8080 }
```

#### 14.8.3 Metrics

> **VERIFY:** confirm whether the API exposes a Prometheus `/metrics` endpoint.
> If present, recommended scrape:

```yaml
- job_name: ida-api
  metrics_path: /metrics
  static_configs:
    - targets: [ "ida-api.ida.svc.cluster.local:8080" ]
```

Recommended dashboards:

* **Request rate / status / latency** (RED method) — `http_requests_total`, `http_request_duration_seconds`, by `path`, `method`, `status`.
* **Database pool** — open / idle / in-use connections.
* **Redis** — command rate, keyspace size, evictions.
* **Chain client** — RPC call rate, RPC error rate, mean tx confirmation time.
* **Auth funnel** — OTP issued / verified / failed; refresh token rotated / replayed.

#### 14.8.4 Log shipping

JSON logs to stdout — pipe into:

* **Cluster:** Fluent Bit / Vector → Loki / Elasticsearch / OpenSearch.
* **Single-host:** `docker compose logs` → `journald` or files mounted at `/var/log/ida/`.

Required fields to keep searchable: `request_id`, `auth_subject`, `auth_role`, `path`, `status`, `latency_ms`.

#### 14.8.5 Tracing

Recommended (not yet wired): OpenTelemetry — wrap Chi router with `otelhttp.NewMiddleware`. Propagate `traceparent` from clients (Portal already sets `X-Request-Id`).

> **VERIFY:** OTel integration status in `pkg/telemetry/`.

***