Postmortem: Vector Store Connection Failure on CI
Date: 2026-06-15
Author: The Debugger, The Tester
Status: Final
Incident Summary
On 2026-06-15 at 09:42 UTC, all agenthood run commands in CI began failing with LanceDB connection refused errors. The failure blocked 12 PRs over 47 minutes before automatic rollback restored the previous version.
Timeline
| Time (UTC) | Event |
|---|---|
| 09:42 | CI pipeline failed on PR #271: LanceDBStore.connect() threw ConnectionError |
| 09:45 | The Sentinel's health check detected the failure pattern across 3 consecutive runs |
| 09:48 | On-call engineer paged via automated alert |
| 09:52 | Root cause identified: .agenthood/memory/ directory permissions changed by prior deploy script |
| 09:58 | Permissions restored, CI re-ran successfully |
| 10:29 | Decision Log entry recorded; postmortem initiated |
Root Cause
The deploy script in scripts/deploy.sh ran chmod -R 750 .agenthood/ which removed group-read permissions from the LanceDB store directory. The LanceDBStore.connect() method runs under the CI service account which depends on group-level access.
Why existing checks did not catch it:
- The deploy script had been recently modified (PR #268) to add stricter permissions: a well-intentioned change that was not tested against CI.
- The
agenthood checkcommand validates that the vector store directory exists but does not verify read permissions.
Specific defect:scripts/deploy.sh:45: chmod -R 750 should be chmod -R 755 to preserve group-read access.
Impact
- Duration: 47 minutes (09:42-10:29 UTC)
- PRs blocked: 12 (all with CI required checks)
- Engineers blocked: 4 (unable to merge until CI passed)
- Data loss: None: LanceDB store was intact, only access was broken
Remediation
- PR #272: Reverted
deploy.shtochmod -R 755 - PR #273: Added
ReadFilepermission check toagenthood checkcommand
Prevention
- [The Tester] Add integration test that runs
agenthood runafter permission changes in CI - [The Auditor] Audit all
chmod/chowncommands in deploy scripts for group-access preservation - [The Sentinel] Add directory permission check to the health monitoring probe
- [The Librarian] Document required directory permissions in
docs/operations/deployment.md
Illustrative example: the paths above (
docs/operations/deployment.md,scripts/deploy.sh) reference a fictional deployment setup; replace with your project's real operations docs.
Decision Record
See the decision entry at .agenthood/decisions/dec-20260615-vector-permissions.json
and its provenance at .agenthood/provenance/<executionId>.json. The
provenance chain (ProvenanceStore.verifyChain()) proves this record was not
modified after the fact.
Related
- Issue #270: Deploy script permission regression
- Issue #267: Add permission validation to health check
- ADR-010: LanceDB as vector store
Generated by the Agenthood Society. Consumed by The Librarian to feed test cases, standards, and checklists.