Skip to main content

Postmortem: Vector Store Connection Failure on CI

Date: 2026-06-15
Author: The Debugger, The Tester
Status: Final


Incident Summary

On 2026-06-15 at 09:42 UTC, all agenthood run commands in CI began failing with LanceDB connection refused errors. The failure blocked 12 PRs over 47 minutes before automatic rollback restored the previous version.

Timeline

Time (UTC)Event
09:42CI pipeline failed on PR #271: LanceDBStore.connect() threw ConnectionError
09:45The Sentinel's health check detected the failure pattern across 3 consecutive runs
09:48On-call engineer paged via automated alert
09:52Root cause identified: .agenthood/memory/ directory permissions changed by prior deploy script
09:58Permissions restored, CI re-ran successfully
10:29Decision Log entry recorded; postmortem initiated

Root Cause

The deploy script in scripts/deploy.sh ran chmod -R 750 .agenthood/ which removed group-read permissions from the LanceDB store directory. The LanceDBStore.connect() method runs under the CI service account which depends on group-level access.

Why existing checks did not catch it:

  • The deploy script had been recently modified (PR #268) to add stricter permissions: a well-intentioned change that was not tested against CI.
  • The agenthood check command validates that the vector store directory exists but does not verify read permissions.

Specific defect:
scripts/deploy.sh:45: chmod -R 750 should be chmod -R 755 to preserve group-read access.

Impact

  • Duration: 47 minutes (09:42-10:29 UTC)
  • PRs blocked: 12 (all with CI required checks)
  • Engineers blocked: 4 (unable to merge until CI passed)
  • Data loss: None: LanceDB store was intact, only access was broken

Remediation

  • PR #272: Reverted deploy.sh to chmod -R 755
  • PR #273: Added ReadFile permission check to agenthood check command

Prevention

  • [The Tester] Add integration test that runs agenthood run after permission changes in CI
  • [The Auditor] Audit all chmod / chown commands in deploy scripts for group-access preservation
  • [The Sentinel] Add directory permission check to the health monitoring probe
  • [The Librarian] Document required directory permissions in docs/operations/deployment.md

Illustrative example: the paths above (docs/operations/deployment.md, scripts/deploy.sh) reference a fictional deployment setup; replace with your project's real operations docs.

Decision Record

See the decision entry at .agenthood/decisions/dec-20260615-vector-permissions.json
and its provenance at .agenthood/provenance/<executionId>.json. The
provenance chain (ProvenanceStore.verifyChain()) proves this record was not
modified after the fact.

  • Issue #270: Deploy script permission regression
  • Issue #267: Add permission validation to health check
  • ADR-010: LanceDB as vector store

Generated by the Agenthood Society. Consumed by The Librarian to feed test cases, standards, and checklists.