Production

This document provides comprehensive instructions for deploying FlowState to production on DigitalOcean infrastructure.

Table of Contents

• Architecture Overview
• Infrastructure Requirements
• Prerequisites
• Initial Setup
• Building Docker Images
• Deployment
• Volume Storage
• Data Encryption
• Service Configuration
• SSL/TLS Setup
• Monitoring & Maintenance
• Troubleshooting

Architecture Overview

Services

Network Topology

Internet
    │
    ▼
┌─────────────────────────────────────────────────────────────┐
│  Kong API Gateway (ports 80/443)                            │
│  - SSL termination                                          │
│  - JWT validation                                           │
│  - Rate limiting                                            │
│  - CORS handling                                            │
└─────────────────────────────────────────────────────────────┘
    │
    ├──► /health      → RxDB Server (public health check)
    ├──► /auth/*      → Auth Server
    ├──► /api/*       → RxDB Server
    ├──► /sync/*      → RxDB Server
    ├──► /ws/*        → RxDB Server (WebSocket)
    ├──► /mcp/*       → MCP HTTP Server
    ├──► /obs/*       → Observability Server
    └──► /ams/*       → Agent Memory Server

Infrastructure Requirements

DigitalOcean Resources

Current Production Configuration

• Droplet: flowstate-prod (nyc3 region)
• IP: 165.227.112.213
• Volume: flowstate-data (100GB, mounted at /mnt/flowstate-data)
• Domain: api.epicflowstate.ai

Prerequisites

1. 1Password CLI

All secrets are managed via 1Password with a service account token. No secrets are stored on the production server except the service account token.

# Install (macOS - for local development)
brew install 1password-cli

# For local development, authenticate interactively
op signin

# For production, use service account token
export OP_SERVICE_ACCOUNT_TOKEN="ops_..."

2. Required Secrets (vault: flowstate-prod)

Production uses the flowstate-prod vault with a service account. All secrets are fetched at runtime.

See docs/SECRETS-REFERENCE.md for detailed requirements and scopes for each secret.

3. SSH Key

Create a dedicated deployment SSH key:

# Generate key
ssh-keygen -t ed25519 -f ~/.ssh/flowstate-deploy -C "flowstate-deploy"

# Add to DigitalOcean
doctl compute ssh-key create flowstate-deploy --public-key "$(cat ~/.ssh/flowstate-deploy.pub)"

4. DigitalOcean CLI

# Install
brew install doctl

# Authenticate
doctl auth init

Initial Setup

1. Create Droplet

# Create droplet with SSH key
doctl compute droplet create flowstate-prod \
  --region nyc3 \
  --size s-2vcpu-4gb \
  --image ubuntu-24-04-x64 \
  --ssh-keys <SSH_KEY_ID> \
  --enable-monitoring \
  --wait

2. Create and Attach Block Storage

# Create 100GB volume
doctl compute volume create flowstate-data \
  --region nyc3 \
  --size 100GiB \
  --desc "FlowState production data volume" \
  --fs-type ext4

# Attach to droplet
doctl compute volume-action attach <VOLUME_ID> <DROPLET_ID> --wait

3. Configure Firewall

# Create firewall
doctl compute firewall create \
  --name flowstate-firewall \
  --inbound-rules "protocol:tcp,ports:22,address:YOUR_IP/32 protocol:tcp,ports:80,address:0.0.0.0/0 protocol:tcp,ports:443,address:0.0.0.0/0" \
  --outbound-rules "protocol:tcp,ports:all,address:0.0.0.0/0 protocol:udp,ports:all,address:0.0.0.0/0"

# Attach to droplet
doctl compute firewall add-droplets <FIREWALL_ID> --droplet-ids <DROPLET_ID>

4. Server Setup

SSH into the server and run initial setup:

ssh -i ~/.ssh/flowstate-deploy root@165.227.112.213

On the server:

# Update system
apt-get update && apt-get upgrade -y

# Install Docker
curl -fsSL https://get.docker.com | sh
systemctl enable docker
systemctl start docker

# Install Docker Compose plugin
apt-get install -y docker-compose-plugin

# Install utilities
apt-get install -y git curl jq htop

# Create deployment directory
mkdir -p /opt/flowstate
mkdir -p /opt/flowstate/.jwt-keys
mkdir -p /opt/flowstate/docker/kong

# Configure Docker log rotation
cat > /etc/docker/daemon.json << 'EOF'
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "5"
  }
}
EOF

systemctl restart docker

5. Mount Block Storage

# Create mount point
mkdir -p /mnt/flowstate-data

# Mount the volume (device name may vary)
mount -o discard,defaults,noatime /dev/disk/by-id/scsi-0DO_Volume_flowstate-data /mnt/flowstate-data

# Add to fstab for persistence
echo "/dev/disk/by-id/scsi-0DO_Volume_flowstate-data /mnt/flowstate-data ext4 defaults,nofail,discard,noatime 0 2" >> /etc/fstab

# Create directory structure
mkdir -p /mnt/flowstate-data/{rxdb-data,rxdb-auth,auth-data,redis-data,surrealdb-data,ollama-data,git-worktrees,worker-logs,worker-claude-config}
chmod -R 755 /mnt/flowstate-data

Building Docker Images

Using the Deployment Script

The recommended way to build and deploy is using the deployment script:

# From the project root
./scripts/deploy-prod.sh build

Manual Build Commands

If building manually, export secrets first:

export GITHUB_TOKEN=$(op read "op://flowstate-development/GITHUB_TOKEN/password")
export RXDB_PREMIUM=$(op read "op://flowstate-development/RXDB_PREMIUM/password")

Build Base Image

The base image contains all pre-built packages:

DOCKER_BUILDKIT=1 docker build \
  --secret id=github_token,env=GITHUB_TOKEN \
  --secret id=rxdb_premium,env=RXDB_PREMIUM \
  --build-arg REPO_BRANCH=dev \
  -f docker/Dockerfile.base \
  -t flowstate-base:latest \
  .

Build Slim Service Images

Slim images are built FROM the base image:

# RxDB Server
docker build -f docker/Dockerfile.rxdb-server.slim -t flowstate-rxdb-server:prod .

# Auth Server
docker build -f docker/Dockerfile.auth-server.slim -t flowstate-auth-server:prod .

# MCP HTTP
docker build -f docker/Dockerfile.mcp-http.slim -t flowstate-mcp-http:prod .

# Obs Server
docker build -f docker/Dockerfile.obs-server.slim -t flowstate-obs-server:prod .

# RAG Sync
docker build -f docker/Dockerfile.rag-sync.slim -t flowstate-rag-sync:prod .

# Orchestrator
docker build -f docker/Dockerfile.orchestrator.slim -t flowstate-orchestrator:prod .

# AMS (Python, standalone)
docker build -f docker/Dockerfile.ams -t flowstate-ams:prod .

Deployment

Security Model

Production uses a secure secrets management model:

• Only one secret on disk: The 1Password service account token (/root/.op-token)
• All other secrets fetched at runtime: From 1Password flowstate-prod vault
• JWT keys written at startup: To memory/tmpfs, fetched from 1Password
• No .env file on server: Environment variables passed at container start time

Initial Droplet Setup (One-time)

From your local machine:

# Set your 1Password service account token in .env
# FLOWSTATE_PROD_OP_TOKEN=ops_...

# Run the setup script
./scripts/prod-setup-droplet.sh

This script:

1. Installs Docker and 1Password CLI on the droplet
2. Transfers the service account token securely
3. Copies scripts and docker configs (no secrets)
4. Configures security hardening

Starting Services (On Server)

SSH into the server and use prod-start.sh:

ssh -i ~/.ssh/flowstate-deploy root@165.227.112.213

# Validate secrets are accessible
/opt/flowstate/scripts/prod-start.sh validate

# Start services (fetches secrets from 1Password, then starts containers)
/opt/flowstate/scripts/prod-start.sh start

# Other commands
/opt/flowstate/scripts/prod-start.sh stop      # Stop services
/opt/flowstate/scripts/prod-start.sh restart   # Restart with fresh secrets
/opt/flowstate/scripts/prod-start.sh logs      # Follow logs
/opt/flowstate/scripts/prod-start.sh status    # Check status
/opt/flowstate/scripts/prod-start.sh secrets   # Show secrets (masked)
/opt/flowstate/scripts/prod-start.sh shell     # Open shell with secrets loaded

How Secrets Flow

┌─────────────────────────────────────────────────────────────────────┐
│  1Password Vault (flowstate-prod)                                   │
│  - JWT keys, API keys, encryption keys, etc.                        │
└─────────────────────────────────────────────────────────────────────┘
                              │
                              │ OP_SERVICE_ACCOUNT_TOKEN
                              ▼
┌─────────────────────────────────────────────────────────────────────┐
│  Droplet (/root/.op-token)                                          │
│  - Only stores the service account token                            │
└─────────────────────────────────────────────────────────────────────┘
                              │
                              │ prod-start.sh start
                              ▼
┌─────────────────────────────────────────────────────────────────────┐
│  prod-fetch-secrets.sh                                              │
│  - Fetches all secrets from 1Password                               │
│  - Writes JWT keys to /opt/flowstate/docker/.jwt-keys/              │
│  - Exports environment variables                                     │
└─────────────────────────────────────────────────────────────────────┘
                              │
                              │ Environment variables
                              ▼
┌─────────────────────────────────────────────────────────────────────┐
│  docker compose -f docker-compose.prod.yml up -d                    │
│  - Containers receive secrets via environment variables             │
│  - No secrets written to docker-compose.yml or .env files           │
└─────────────────────────────────────────────────────────────────────┘

Re-encrypting LUKS Volume (New Secrets)

If you need to re-encrypt the data volume with new keys:

# From local machine - WARNING: DESTROYS ALL DATA
./scripts/prod-reencrypt-volume.sh

This will:

1. Stop all services
2. Fetch new LUKS key from 1Password
3. Re-format the volume with LUKS2
4. Create fresh directory structure
5. Configure auto-unlock on boot

Legacy Manual Deployment (Deprecated)

Note: The following manual steps are deprecated. Use prod-setup-droplet.sh and prod-start.sh instead.

<details> <summary>Click to expand deprecated manual steps</summary>

1. Sync files to server:

scp -i ~/.ssh/flowstate-deploy -r docker/docker-compose.prod.yml docker/kong root@165.227.112.213:/opt/flowstate/docker/

2. Create .env file on server (DEPRECATED - use 1Password instead):

# DO NOT DO THIS - secrets should come from 1Password
# cat > /opt/flowstate/docker/.env << EOF
# ...secrets...
# EOF

3. Generate JWT keys (DEPRECATED - stored in 1Password):

# Keys are now fetched from 1Password at startup
# No need to generate or store keys on server

4. Start services:

cd /opt/flowstate/docker
docker compose -f docker-compose.prod.yml up -d

Volume Storage

All persistent data is stored on the attached block storage volume:

Mount Point

/mnt/flowstate-data/
├── rxdb-data/          # RxDB database files
├── rxdb-auth/          # RxDB authentication data
├── auth-data/          # Auth server persistent data
├── redis-data/         # Redis AOF persistence
├── surrealdb-data/     # SurrealDB vector database
├── ollama-data/        # Ollama model cache
├── git-worktrees/      # Git worktrees for agents
├── worker-logs/        # Worker container logs
└── worker-claude-config/ # Worker Claude configurations

Volume Mapping in docker-compose.prod.yml

services:
  rxdb-server:
    volumes:
      - /mnt/flowstate-data/rxdb-data:/app/data
      - /mnt/flowstate-data/rxdb-auth:/data/auth

  redis:
    volumes:
      - /mnt/flowstate-data/redis-data:/data

  surrealdb:
    volumes:
      - /mnt/flowstate-data/surrealdb-data:/data

  ollama:
    volumes:
      - /mnt/flowstate-data/ollama-data:/root/.ollama

  # ... etc

Backup Recommendations

# Create snapshot of the volume
doctl compute volume-action create-snapshot flowstate-data --snapshot-name "flowstate-backup-$(date +%Y%m%d)"

# Or backup specific directories
tar -czf /tmp/flowstate-backup.tar.gz /mnt/flowstate-data/rxdb-data /mnt/flowstate-data/redis-data

Data Encryption

FlowState uses multiple layers of encryption to protect data at rest:

Encryption Layers

LUKS Volume Encryption

The production volume is encrypted using LUKS2 with AES-XTS cipher (512-bit key). This provides:

• Full disk encryption: All data written to the volume is encrypted
• Automatic unlock on boot: Using a key file stored on the root filesystem
• Transparent to applications: Services read/write as normal; encryption is handled at the block layer

LUKS Configuration

# Check LUKS status
cryptsetup status flowstate-crypt

# View LUKS header info
cryptsetup luksDump /dev/disk/by-id/scsi-0DO_Volume_flowstate-data

Key Management

The LUKS key file is stored at /root/.luks-keyfile with mode 600 (root-only access).

IMPORTANT: The key file is also stored in 1Password vault flowstate-development:

• Item: LUKS_KEYFILE
• Field: keyfile (base64 encoded)

To restore the key file from 1Password:

# Decode and save key file
op read "op://flowstate-development/LUKS_KEYFILE/keyfile" | base64 -d > /root/.luks-keyfile
chmod 600 /root/.luks-keyfile

Boot Configuration

Auto-unlock is configured via:

• /etc/crypttab:

  flowstate-crypt /dev/disk/by-id/scsi-0DO_Volume_flowstate-data /root/.luks-keyfile luks
  

• /etc/fstab:

  /dev/mapper/flowstate-crypt /mnt/flowstate-data ext4 defaults,nofail 0 2
  

Application-Level Encryption Keys

Additional encryption keys are available for future application-level encryption:

These keys are stored in:

• Server: /opt/flowstate/docker/.env
• 1Password: flowstate-development vault

Emergency Key Recovery

If the server is destroyed but the volume survives:

1. Attach volume to new droplet
2. Install cryptsetup: apt-get install cryptsetup
3. Retrieve key from 1Password and save to /root/.luks-keyfile
4. Open encrypted volume: cryptsetup luksOpen --key-file /root/.luks-keyfile /dev/disk/by-id/scsi-0DO_Volume_flowstate-data flowstate-crypt
5. Mount: mount /dev/mapper/flowstate-crypt /mnt/flowstate-data

Setting Up LUKS Encryption (New Deployment)

WARNING: This destroys all existing data on the volume!

# 1. Stop all services
cd /opt/flowstate/docker
docker compose -f docker-compose.prod.yml down

# 2. Unmount volume
umount /mnt/flowstate-data

# 3. Generate key file (or retrieve from 1Password)
dd if=/dev/urandom of=/root/.luks-keyfile bs=4096 count=1
chmod 600 /root/.luks-keyfile

# 4. Format with LUKS
cryptsetup luksFormat --type luks2 --key-file /root/.luks-keyfile /dev/disk/by-id/scsi-0DO_Volume_flowstate-data --batch-mode

# 5. Open encrypted volume
cryptsetup luksOpen --key-file /root/.luks-keyfile /dev/disk/by-id/scsi-0DO_Volume_flowstate-data flowstate-crypt

# 6. Create filesystem
mkfs.ext4 /dev/mapper/flowstate-crypt

# 7. Mount and create directories
mount /dev/mapper/flowstate-crypt /mnt/flowstate-data
mkdir -p /mnt/flowstate-data/{rxdb-data,rxdb-auth,auth-data,redis-data,surrealdb-data,ollama-data,git-worktrees,worker-logs,worker-claude-config}
chmod -R 777 /mnt/flowstate-data/{rxdb-data,rxdb-auth,auth-data,redis-data,surrealdb-data,ollama-data,worker-logs}

# 8. Configure auto-unlock
echo 'flowstate-crypt /dev/disk/by-id/scsi-0DO_Volume_flowstate-data /root/.luks-keyfile luks' >> /etc/crypttab
echo '/dev/mapper/flowstate-crypt /mnt/flowstate-data ext4 defaults,nofail 0 2' >> /etc/fstab

# 9. Start services
docker compose -f docker-compose.prod.yml up -d

# 10. IMPORTANT: Save key to 1Password!
base64 < /root/.luks-keyfile | pbcopy  # Copy to clipboard, save to 1Password

Service Configuration

Kong Gateway

Kong configuration is in docker/kong/kong.prod.yml:

• JWT Validation: Validates tokens using RSA public key
• CORS: Configured for production domains (epicflowstate.ai, flowstate.dev, etc.)
• Rate Limiting: 1000 requests/minute
• HTTP/2: Enabled for multiplexed connections (required for 29+ collection streams)
• Routes:
  • /health → RxDB Server (public, no auth - for health checks)
  • /auth/* → Auth Server
  • /api/*, /sync/* → RxDB Server
  • /ws/* → RxDB WebSocket
  • /mcp/* → MCP HTTP Server
  • /obs/* → Observability Server
  • /ams/* → Agent Memory Server

SSE over HTTP/2 Configuration

The RxDB sync uses Server-Sent Events (SSE) for real-time pullStream endpoints. SSE over HTTP/2 requires special configuration:

nginx-http.conf (included via KONG_NGINX_HTTP_INCLUDE):

# HTTP/2 and SSE optimizations for Kong
proxy_connect_timeout 60s;
proxy_send_timeout 24h;
proxy_read_timeout 24h;

# Disable buffering for SSE (critical for streaming)
proxy_buffering off;
proxy_request_buffering off;

# Disable gzip for SSE - compression breaks streaming
gzip off;

# Allow large headers for JWT tokens
large_client_header_buffers 4 32k;

Service timeouts in kong.yml (rxdb-service):

- name: rxdb-service
  url: http://rxdb-server:3002
  connect_timeout: 60000      # 60 seconds
  read_timeout: 86400000      # 24 hours for SSE
  write_timeout: 86400000     # 24 hours for SSE

Important: HTTP/2 is required because browsers limit HTTP/1.1 to 6 concurrent connections per origin. With 29+ collections each needing a pullStream connection, HTTP/1.1 would cause connection exhaustion.

Updating Kong Configuration

After modifying kong.yml or kong.prod.yml:

# Validate configuration
docker exec flowstate-kong kong config parse /kong/kong.yml

# Reload Kong (applies changes without restart)
docker exec flowstate-kong kong reload

Note: The kong.yml file is the active configuration. If you edit kong.prod.yml, copy it to kong.yml:

cp /opt/flowstate/docker/kong/kong.prod.yml /opt/flowstate/docker/kong/kong.yml
docker exec flowstate-kong kong reload

Environment Variables

Key environment variables for services:

Generating Service JWT

For the orchestrator to communicate with RxDB:

# On server
PRIVATE_KEY_FILE="/opt/flowstate/.jwt-keys/private.pem"
CURRENT_TIME=$(date +%s)
EXPIRY_TIME=$((CURRENT_TIME + 315360000))  # 10 years

HEADER=$(echo -n '{"alg":"RS256","typ":"JWT"}' | base64 -w0 | tr '+/' '-_' | tr -d '=')
PAYLOAD=$(echo -n "{\"iss\":\"https://api.epicflowstate.ai\",\"sub\":\"orchestrator-service\",\"domainId\":\"flowstate-prod\",\"orgId\":\"org_9f3omFEY2H\",\"iat\":$CURRENT_TIME,\"exp\":$EXPIRY_TIME,\"role\":\"service\"}" | base64 -w0 | tr '+/' '-_' | tr -d '=')
SIGNATURE=$(echo -n "${HEADER}.${PAYLOAD}" | openssl dgst -sha256 -sign "$PRIVATE_KEY_FILE" | base64 -w0 | tr '+/' '-_' | tr -d '=')

echo "${HEADER}.${PAYLOAD}.${SIGNATURE}"

SSL/TLS Setup

Option 1: Cloudflare (Recommended)

1. Add domain to Cloudflare
2. Point A record to droplet IP
3. Enable "Full (strict)" SSL mode
4. Cloudflare handles SSL termination

Option 2: Let's Encrypt with Certbot

1. Install certbot:

apt-get install -y certbot

2. Stop Kong temporarily (to free port 80):

cd /opt/flowstate/docker
docker compose -f docker-compose.prod.yml stop kong

3. Generate certificate:

certbot certonly --standalone -d api.epicflowstate.ai

4. Copy certificates to Kong directory:

mkdir -p /opt/flowstate/docker/kong/ssl
cp /etc/letsencrypt/live/api.epicflowstate.ai/fullchain.pem /opt/flowstate/docker/kong/ssl/
cp /etc/letsencrypt/live/api.epicflowstate.ai/privkey.pem /opt/flowstate/docker/kong/ssl/
chmod 644 /opt/flowstate/docker/kong/ssl/*.pem

5. Start Kong:

docker compose -f docker-compose.prod.yml up -d kong

6. Set up auto-renewal hook (copies new certs and reloads Kong):

cat > /etc/letsencrypt/renewal-hooks/deploy/kong-ssl.sh << 'EOF'
#!/bin/bash
# Copy renewed certificates to Kong directory
cp /etc/letsencrypt/live/api.epicflowstate.ai/fullchain.pem /opt/flowstate/docker/kong/ssl/
cp /etc/letsencrypt/live/api.epicflowstate.ai/privkey.pem /opt/flowstate/docker/kong/ssl/
chmod 644 /opt/flowstate/docker/kong/ssl/*.pem

# Reload Kong to pick up new certificates
docker exec flowstate-kong kong reload
EOF

chmod +x /etc/letsencrypt/renewal-hooks/deploy/kong-ssl.sh

7. Test renewal (dry run):

certbot renew --dry-run

Note: Certbot automatically sets up a systemd timer that runs twice daily to check for certificate renewal.

Monitoring & Maintenance

Health Checks

# Check all service health
docker compose -f docker-compose.prod.yml ps

# Check specific service
docker inspect --format='{{.State.Health.Status}}' flowstate-rxdb-server

Viewing Logs

# All services
docker compose -f docker-compose.prod.yml logs -f

# Specific service
docker compose -f docker-compose.prod.yml logs -f rxdb-server

# Last 100 lines
docker compose -f docker-compose.prod.yml logs --tail=100

Resource Monitoring

# Container stats
docker stats

# Disk usage
df -h /mnt/flowstate-data

# Memory usage
free -h

Pulling Ollama Models

# Pull embedding model
docker exec flowstate-ollama ollama pull nomic-embed-text

# List installed models
docker exec flowstate-ollama ollama list

Updating Services

# Pull latest images
docker compose -f docker-compose.prod.yml pull

# Recreate containers
docker compose -f docker-compose.prod.yml up -d --force-recreate

# Or update specific service
docker compose -f docker-compose.prod.yml up -d --force-recreate rxdb-server

Troubleshooting

Common Issues

Services Not Starting

# Check logs for errors
docker compose -f docker-compose.prod.yml logs <service-name>

# Check if dependencies are healthy
docker compose -f docker-compose.prod.yml ps

Kong Configuration Errors

# Validate Kong config
docker exec flowstate-kong kong check /kong/kong.yml

# Reload Kong config
docker exec flowstate-kong kong reload

Volume Permission Issues

# SurrealDB needs root user
# Ensure docker-compose has: user: root

# Check directory permissions
ls -la /mnt/flowstate-data/

SSH Connection Issues

# Check firewall
doctl compute firewall list

# Update allowed IPs
doctl compute firewall update <FIREWALL_ID> \
  --inbound-rules "protocol:tcp,ports:22,address:YOUR_NEW_IP/32 ..."

Out of Disk Space

# Check volume usage
df -h /mnt/flowstate-data

# Clean Docker resources
docker system prune -a

# Check large files
du -sh /mnt/flowstate-data/* | sort -h

Service-Specific Issues

ERR_HTTP2_PROTOCOL_ERROR on pullStream Endpoints

If browsers show net::ERR_HTTP2_PROTOCOL_ERROR 200 on /sync/*/pullStream endpoints:

1. Verify gzip is disabled in nginx-http.conf:

cat /opt/flowstate/docker/kong/nginx-http.conf | grep gzip
# Should show: gzip off;

2. Check service timeouts in kong.yml:

grep -A4 'name: rxdb-service' /opt/flowstate/docker/kong/kong.yml
# Should show read_timeout: 86400000 (24 hours)

3. Verify buffering is disabled:

grep proxy_buffering /opt/flowstate/docker/kong/nginx-http.conf
# Should show: proxy_buffering off;

4. Reload Kong after any config changes:

docker exec flowstate-kong kong reload

Note: The 200 status in the error indicates the initial response succeeded but the stream broke during transfer. This is typically caused by gzip compression or proxy buffering on SSE streams.

Auth Server Routes Not Working

If requests to /auth/* endpoints return "Cannot POST /auth/send-code" or similar:

1. Check logs for route mounting:

docker logs flowstate-auth-server --tail=20

Look for these log messages:

• ✅ "Storage initialized" - Storage adapter loaded
• ✅ "All routes mounted" - Routes registered successfully
• ❌ Only "Auth server listening" - Routes failed to mount (check JWT keys)

2. Verify JWT keys are accessible:

# Check keys exist in container
docker exec flowstate-auth-server ls -la /app/jwt-keys/

# Keys must be readable (644 permissions)
# If empty or permission denied, copy keys and fix permissions:
cp /opt/flowstate/.jwt-keys/*.pem /opt/flowstate/docker/.jwt-keys/
chmod 644 /opt/flowstate/docker/.jwt-keys/*.pem

# Recreate container to pick up new volume contents
cd /opt/flowstate/docker
docker compose -f docker-compose.prod.yml up -d auth-server

3. Check SendGrid configuration (if emails fail):

# Verify MAIL_FROM is set
docker exec flowstate-auth-server env | grep MAIL_FROM

# If missing, add to .env and recreate container:
echo 'MAIL_FROM=your-verified-email@domain.com' >> /opt/flowstate/docker/.env
docker compose -f docker-compose.prod.yml up -d auth-server

RxDB Server

# Check database health
curl http://localhost:80/api/health

# View detailed logs
docker logs flowstate-rxdb-server --tail=100

Orchestrator Missing Environment Variables

The orchestrator requires:

• ORG_ID - FlowState organization ID
• RXDB_AUTH_TOKEN - Valid JWT for RxDB authentication

# Verify environment
docker exec flowstate-orchestrator env | grep -E "ORG_ID|RXDB_AUTH_TOKEN"

SurrealDB Health Check Failing

SurrealDB uses a minimal image without curl. The healthcheck must use the native CLI:

healthcheck:
  test: ['CMD', '/surreal', 'isready', '--endpoint', 'http://localhost:8000']

Ollama Health Check Failing

Ollama doesn't have curl. Use the native CLI:

healthcheck:
  test: ['CMD', 'ollama', 'list']

Emergency Recovery

Restart All Services

docker compose -f docker-compose.prod.yml down
docker compose -f docker-compose.prod.yml up -d

Restore from Volume Snapshot

# List snapshots
doctl compute snapshot list --resource volume

# Restore (requires detaching current volume first)
doctl compute volume-action detach <VOLUME_ID> <DROPLET_ID>
doctl compute volume create-from-snapshot <SNAPSHOT_ID> --name flowstate-data-restored
doctl compute volume-action attach <NEW_VOLUME_ID> <DROPLET_ID>

Complete Rebuild

# On server
cd /opt/flowstate/docker
docker compose -f docker-compose.prod.yml down -v
docker system prune -a

# Re-deploy from local machine
./scripts/deploy-prod.sh deploy

Quick Reference

Important Paths

Important Commands

# Deploy
./scripts/deploy-prod.sh deploy

# Check status
./scripts/deploy-prod.sh status

# View logs
./scripts/deploy-prod.sh logs -f

# Restart
./scripts/deploy-prod.sh restart

# SSH to server
ssh -i ~/.ssh/flowstate-deploy root@165.227.112.213

Service URLs (via Kong)

Built with Epic Flowstate