Engineering report: pgwd v1.1.1

pgwd v1.1.1 product infographic

pgwd (Postgres Watch Dog) is a Go CLI that samples PostgreSQL connection pressure and notifies before the instance runs out of slots. It does not replace a pooler such as PgBouncer. It is a specialist watchdog: visibility and alerts, then you act.

v1.1.1 (2026-08-31) is a security patch on the 1.0 stable API. No breaking config changes in this tag. Operators should pull the binary / GHCR image so they pick up Go 1.26.6 and golang.org/x/net v0.56.0.

1. Why connection slots matter

In high-availability designs, PostgreSQL connection slots are a scarce, volatile resource. Exhausting them surfaces as SQLSTATE 53300 (too many clients already). Available slots are not simply max_connections: PostgreSQL reserves capacity for privileged roles via superuser_reserved_connections and, from PostgreSQL 16+, reserved_connections.

When slots are gone, application traffic is already blocked. That state often coincides with CPU, I/O, and memory pressure (shared_buffers, work_mem) and can precede OOM. pgwd exists so on-call sees Attention / Alert / Danger (and stale sessions) before that cliff.

2. Data flow and metrics backends

A check cycle is:

  1. Check — query pg_stat_activity (active, idle, backend start times, optional long-running queries).
  2. Persist — write history used for hysteresis, resolution alerts, and /metrics.
  3. Emit — optional CSV export; optional HTTP /metrics (Prometheus) and /healthz.
Backend Prefer when Architect note
SQLite (default) Local / single-node monitor Recommended. Avoids a circular dependency: if the network to Postgres is down, the watchdog can still record locally.
Postgres / Timescale Fleet-scale history Centralize metrics in SQL.
MySQL Heterogeneous shops already on MySQL for metrics Extra network dependency; use only if that is the org standard.

SQLite (and the SQL store) keys rows by (client, cluster, database), not by URL host. Give each databases: entry a unique client when the same database name exists on different hosts.

Kubernetes: kube.postgres / -kube-postgres is client-go port-forward, not a kubectl subprocess — that keeps the distroless image small. Multi-entry databases: cannot be combined with kube postgres in one process (a single databases: entry + kube is supported).

3. Detection and notification

Stale connections

Stale detection uses session age (backend start) against a configured minimum age and a stale count threshold ≥ 1. Long-lived idle sessions fragment memory and occupy slots that should be free for new work.

Three-level thresholds and noise suppression

Utilization vs max_connections uses percentage levels (defaults in the 75 / 85 / 95 band: Attention / Alert / Danger), plus idle and stale thresholds. Only the highest level reached in a cycle fires — that avoids alert storms when load ramps through several bands in one interval.

From v1.1.0, threshold notifiers are quiet by default: they fire on transition, escalation, and de-escalation, not every interval while the bad state persists. Set notifications.repeat_while_firing (or the matching flag / env) for v1.0-style per-interval repeats. PagerDuty uses a stable dedup_key and event_action: resolve on recovery.

Connect failure always notifies (when notifiers are configured and not -dry-run). There is no opt-in flag. A “too many clients already” connect error is classified as too_many_clients.

Channels: Slack, Grafana Loki, PagerDuty, Microsoft Teams, generic webhook (optional HMAC-SHA256). -test-max-connections overrides the limit inside the monitor only so you can exercise Danger without changing the real Postgres GUC.

4. Supply chain (1.0+) and v1.1.1 security

5. Multi-database and 1.0 breaking changes

databases: is required (even for one target). The top-level db: key is gone — loading it fails with a migration hint. See UPGRADE-0.9-to-1.0.

Fixed total / active count thresholds are gone. Use -db-threshold-levels (percentage bands). notify_on_connect_failure is gone because connect-failure alerts are always on.

Exit codes (automation)

Code Event DBRE note
0 Success Normal check.
1 Config / validation Fail the deploy (CI).
2 Postgres connect failure Single-target: process exits 2. Multi-database: log, skip that target, continue.
3 Stats / query failure One-shot; daemon does not exit on query errors.
4 Notifier failure With --strict.

6. Repositories

Repo Role
hrodrig/pgwd CLI, checker, notifiers, GHCR image, releases.
hrodrig/pgwd-selfhosted Compose, Helm, run/ operator layout. Not the Go engine.

Operator examples: pgwd-selfhosted/run. Full changelog: CHANGELOG.md.