13 Monitoring and Troubleshooting

This chapter covers what to watch on a running instance, where to find logs, and how to fix the failures operators meet most often.

13.1 Health checks

13.1.1 The health endpoint

GET /api/health needs no sign-in and reports whether the application can reach its database:

The endpoint reveals nothing else and is never cached. Point your external monitoring at the public address, for example https://grc.example.com/api/health, so it also covers DNS, the proxy and the certificate.

13.1.2 Container health checks

docker compose ps shows the state of each service. restart: unless-stopped restarts db, web, worker and caddy if they exit. Docker does not restart a container that is only unhealthy.

13.1.3 The audit chain

Verify the audit log's hash chain daily from cron, so a break is reported rather than found later:

30 3 * * * cd /srv/grc && docker compose exec -T web ./node_modules/.bin/tsx prisma/audit-verify.ts >> /var/log/grc-audit-verify.log 2>&1 || echo "GRC audit chain verification failed" | mail -s "GRC audit chain" ops@example.com

The command prints the verdict as JSON and exits 0 when the chain is intact, 1 when it is broken and 2 on an error. Each run is itself recorded in the audit log. Administrators can run the same check under Settings → Audit Log (/settings/audit-log); see Audit Log and Evidence Integrity.

To detect a rewrite by someone holding the database owner's password, also copy the chain head off the server after each verification:

docker compose exec -T db sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -tAc "SELECT \"lastSeq\", \"lastHash\" FROM audit_chain_head"'

A later rewrite no longer matches a hash recorded elsewhere.

13.1.4 What else to watch

13.2 Logs

Each service writes to standard output; read it with docker compose logs:

docker compose logs --tail 200 web
docker compose logs -f worker
docker compose logs migrate
docker compose logs caddy
docker compose logs db

Docker's default log driver keeps container logs without limit. Set a size limit in /etc/docker/daemon.json, for example:

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "50m", "max-file": "5" }
}

Restart Docker and recreate the containers for it to apply. If your organization collects logs centrally, ship the container logs there; the audit log inside the application remains the record of user activity.

13.3 Row counts

deploy/counts.sql prints the row count of every table in the application and queue schemas, one schema.table count line each. Take it before and after any maintenance you do by hand, and compare:

docker compose exec -T db sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -At -F " " -f -' \
  < deploy/counts.sql | sort > counts-$(date -u +%Y%m%dT%H%M%SZ).txt

13.4 Common problems

13.4.1 The stack does not start: a variable is not set

Compose prints Set APP_DB_PASSWORD in .env (or AUTH_SECRET, APP_URL, POSTGRES_PASSWORD) and starts nothing. Add the value to .env and run the command again.

13.4.2 migrate fails

web and worker stay down until migrate succeeds. Read its log:

docker compose logs migrate

To inspect migration state without starting the application, run a command in a one-off migrate container:

docker compose run --rm --no-deps migrate ./node_modules/.bin/prisma migrate status

13.4.3 web is unhealthy or keeps restarting

docker compose ps web
docker compose logs --tail 100 web
curl -s http://127.0.0.1:3000/api/health

13.4.4 Caddy does not get a certificate

docker compose logs caddy shows the ACME error. Check that DOMAIN resolves to this server from the internet, that ports 80 and 443 reach Caddy (no other service holds them, and no firewall blocks them), and that DOMAIN and ACME_EMAIL are set. For a server Let's Encrypt cannot reach, use your own certificate (see Installation).

13.4.5 Email is not sent

1. In Settings → Integrations, click Send test email. The error it shows is the SMTP server's answer.
2. If the test works but notifications do not arrive, check that the worker is running (docker compose ps worker) and read its log for [mailer] delivery <id> failed: lines. Email is sent only by the worker.
3. A configuration that sends a username and password requires STARTTLS (or implicit TLS) from the relay; a relay that does not offer it is refused rather than sent the password in clear.
4. If the web log shows settings.decrypt_failed for smtp.pass, the stored password was encrypted with a different key (after a change of AUTH_SECRET or SETTINGS_ENCRYPTION_KEY, or a restore from another instance). Enter the password again under Settings → Integrations.
5. Digests arriving at the wrong time after a schedule change: the cron is in UTC. If saving the schedule warned that it takes effect when the worker next starts, restart the worker (docker compose restart worker).

Notification setup and the delivery list are described in Notifications and Email.

13.4.6 Single sign-on errors

A refused LDAP or SAML sign-in shows the user a general message and records the reason in the audit log. Look up the sign-in failure entry under Settings → Audit Log, and see Authentication and Single Sign-On for each reason. On the server side, check that APP_URL uses https:// and matches the address registered with the identity provider, and that the server clock is correct: SAML responses older than 10 minutes are refused.

13.4.7 Disk full

A full disk stops PostgreSQL, and with it the application. Find what is using the space:

df -h
docker system df
du -sh /var/backups/grc

Free space by, in order of safety:

1. Moving older local backups off the server, or shortening the local retention period.
2. Removing old rollback images and the build cache (see Upgrades, Cleaning up after upgrades).
3. Limiting container log size (see Logs).
4. Deleting in-app snapshots under Settings → Backups, if they are copied elsewhere.

Never remove a Docker volume to free space. After space is freed, run docker compose --profile tls up -d and check health.

13.4.8 Audit chain verification fails

A broken chain is evidence that an audit entry was changed, deleted or reordered outside the application, which requires the database owner's password or root on the server. The verdict names the first entry that fails and the reason:

Do this:

1. Do not repair anything, and stop further writes if you can (docker compose stop web worker).
2. Take a server backup into a separate directory and preserve the database volume as it is.
3. Record the verdict and the time, and start your incident process. Compare with the chain heads you copied off the server to find when it changed.
4. Restoring an earlier backup brings back an intact trail from that time; decide that as part of the investigation, after the evidence is kept.

13.4.9 Locked out of administrator accounts

Sign in with a break-glass administrator; it keeps password sign-in when SSO only is on and the identity provider is down. Any administrator can reset another user's password under Settings → Users. Repeated failed sign-ins block the address that failed, never the account, so the right password from another network still works. There is no command-line password reset, which is why the post-installation checklist asks for a second break-glass administrator.