Production Security

This document details the enterprise-grade security measures implemented on the FlowState production infrastructure.

Security Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                        INTERNET                                  │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  CLOUD LAYER (DigitalOcean)                                      │
│  └─ DO Cloud Firewall (SSH IP-restricted, 80/443 public)        │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  NETWORK LAYER                                                   │
│  ├─ UFW Firewall (deny all, allow 80/443 only)                  │
│  ├─ Fail2Ban (SSH brute force protection)                       │
│  └─ Tailscale (encrypted mesh VPN for admin access)             │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  HOST LAYER                                                      │
│  ├─ SSH Hardening (key-only, no root password)                  │
│  ├─ AIDE (file integrity monitoring)                            │
│  ├─ Auditd (security event logging)                             │
│  └─ Kernel Hardening (sysctl security settings)                 │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  CONTAINER LAYER                                                 │
│  ├─ Docker (no-new-privileges, seccomp, AppArmor)               │
│  ├─ Secrets (/run/secrets/* file-based)                         │
│  └─ Vector (centralized log aggregation)                        │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  DATA LAYER                                                      │
│  ├─ LUKS2 Volume Encryption (AES-XTS-512bit)                    │
│  ├─ 1Password (runtime secret fetching)                         │
│  └─ JWT Keys (600 permissions, ephemeral)                       │
└─────────────────────────────────────────────────────────────────┘

1. Network Security

DigitalOcean Cloud Firewall

An external cloud firewall provides the first line of defense before traffic reaches the droplet.

Firewall ID: 57514b84-89cc-4d5c-80d8-f85361bb6ec5

Dashboard: https://cloud.digitalocean.com/networking/firewalls/57514b84-89cc-4d5c-80d8-f85361bb6ec5/rules

Inbound Rules:

Outbound Rules:

Key Benefits:

• SSH access restricted to specific admin IPs at the cloud level
• Malicious SSH traffic never reaches the droplet
• Reduces attack surface before UFW/Fail2Ban even see traffic

Management:

# List firewalls via doctl
doctl compute firewall list

# View specific firewall rules
doctl compute firewall get 57514b84-89cc-4d5c-80d8-f85361bb6ec5

# Add SSH IP (use dashboard for complex changes)
doctl compute firewall add-rules 57514b84-89cc-4d5c-80d8-f85361bb6ec5 \
  --inbound-rules "protocol:tcp,ports:22,address:<NEW_IP>/32"

UFW Firewall

The firewall is configured to deny all incoming traffic except explicitly allowed ports.

Current Rules:

Status: active

To                         Action      From
--                         ------      ----
Anywhere on tailscale0     ALLOW       Anywhere       # Admin access
80/tcp                     ALLOW       Anywhere       # HTTP (redirects to HTTPS)
443/tcp                    ALLOW       Anywhere       # HTTPS (Kong API Gateway)

Management Commands:

# View status
ufw status verbose

# Add rule
ufw allow <port>/tcp

# Remove rule
ufw delete allow <port>/tcp

Fail2Ban

Protects SSH against brute force attacks.

Configuration: /etc/fail2ban/jail.local

[DEFAULT]
bantime = 1h
findtime = 10m
maxretry = 3
ignoreip = 127.0.0.1/8 100.64.0.0/10  # Tailscale range

[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 3600

Management Commands:

# Check status
fail2ban-client status sshd

# Unban IP
fail2ban-client set sshd unbanip <IP>

# View banned IPs
fail2ban-client status sshd | grep "Banned IP"

Tailscale VPN

All administrative access is restricted to the Tailscale network.

Server Tailscale IP: 100.113.155.55

Access:

# SSH via Tailscale (only method)
ssh root@100.113.155.55

# Public IP SSH is BLOCKED
ssh root@165.227.112.213  # Will fail

2. Host Security

SSH Hardening

Configuration: /etc/ssh/sshd_config.d/hardening.conf

PermitRootLogin prohibit-password
PasswordAuthentication no
PubkeyAuthentication yes
PermitEmptyPasswords no
X11Forwarding no
MaxAuthTries 3
ClientAliveInterval 300
ClientAliveCountMax 2
AllowAgentForwarding no
AllowTcpForwarding no

AIDE (Advanced Intrusion Detection Environment)

File integrity monitoring for critical system files.

Monitored Paths: /etc/aide/aide.conf.d/99_local_rules

/root/.op-token$ CONTENT_EX
/root/.luks-keyfile$ CONTENT_EX
/opt/flowstate/docker/.jwt-keys CONTENT_EX
/opt/flowstate/scripts CONTENT_EX
/etc/ssh CONTENT_EX
/etc/docker CONTENT_EX

Management Commands:

# Initialize database (first time)
aideinit

# Check for changes
aide --check

# Update database after legitimate changes
aide --update
mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

Daily Checks: /etc/cron.daily/aide-check

Auditd

Security event auditing for compliance and forensics.

Audit Rules: /etc/audit/rules.d/security.rules

# Monitor authentication
-w /etc/passwd -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k sudoers

# Monitor SSH
-w /etc/ssh/sshd_config -p wa -k sshd

# Monitor secrets
-w /root/.op-token -p rwa -k secrets
-w /root/.luks-keyfile -p rwa -k secrets
-w /opt/flowstate/docker/.jwt-keys -p rwa -k secrets

# Monitor Docker
-w /etc/docker -p wa -k docker
-w /var/run/docker.sock -p rwa -k docker

Query Logs:

# Search by key
ausearch -k secrets

# Recent events
ausearch -ts recent

# Generate report
aureport --summary

Kernel Hardening

Configuration: /etc/sysctl.d/99-security.conf

# IP Spoofing protection
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Ignore ICMP broadcast requests
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Disable source packet routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0

# Ignore send redirects
net.ipv4.conf.all.send_redirects = 0

# Block SYN attacks
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 2

# Log Martians
net.ipv4.conf.all.log_martians = 1

# Ignore ICMP redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0

3. Container Security

Docker Daemon Security

Configuration: /etc/docker/daemon.json

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "5"
  },
  "live-restore": true,
  "no-new-privileges": true
}

Security Features:

Docker Secrets

Secrets are stored as files in /run/secrets/ with 600 permissions.

Available Secrets:

/run/secrets/
├── anthropic_api_key
├── github_token
├── openai_api_key
├── redis_encryption_key
├── rxdb_auth_token
├── rxdb_encryption_key
├── rxdb_premium
└── sendgrid_api_key

How Secrets Flow:

1Password (flowstate-prod vault)
         │
         ▼ OP_SERVICE_ACCOUNT_TOKEN
/root/.op-token
         │
         ▼ prod-fetch-secrets.sh
Environment Variables (in memory)
         │
         ▼ write-docker-secrets.sh
/run/secrets/* (file-based, 600 perms)
         │
         ▼ docker-compose
Container Environment Variables

Vector Log Aggregation

Centralized logging for all containers and host logs.

Configuration: /etc/vector/vector.yaml

sources:
  docker_logs:
    type: docker_logs
    docker_host: unix:///var/run/docker.sock

  host_logs:
    type: file
    include:
      - /var/log/auth.log
      - /var/log/syslog
      - /var/log/aide/*.log

sinks:
  file_logs:
    type: file
    path: /var/log/vector/flowstate-%Y-%m-%d.log
    encoding:
      codec: json

Log Location: /var/log/vector/

Management:

# View status
systemctl status vector

# View logs
tail -f /var/log/vector/flowstate-$(date +%Y-%m-%d).log | jq .

# Restart
systemctl restart vector

4. Data Security

LUKS Volume Encryption

The data volume is encrypted with LUKS2.

Encryption Details:

Key Management:

• Keyfile: /root/.luks-keyfile (600 permissions)
• Backup: 1Password vault flowstate-prod → LUKS_KEYFILE
• Auto-unlock: Configured in /etc/crypttab

Commands:

# Check status
cryptsetup status flowstate-crypt

# View LUKS info
cryptsetup luksDump /dev/disk/by-id/scsi-0DO_Volume_flowstate-data

1Password Integration

All secrets are fetched from 1Password at runtime.

Vault: flowstate-prod

Secrets Stored:

Token Location: /root/.op-token (600 permissions)

JWT Key Security

JWT keys are fetched from 1Password and written to disk at service startup.

Location: /opt/flowstate/docker/.jwt-keys/

-rw------- private.pem  # 600 - root only
-rw-r--r-- public.pem   # 644 - readable by services

5. Access Control Summary

Network Access

Administrative Access

6. Incident Response

Suspected Breach

1. Isolate: Disconnect from Tailscale

   tailscale down
   

2. Preserve Evidence:

   # Snapshot volume
   doctl compute volume-action create-snapshot flowstate-data \
     --snapshot-name "incident-$(date +%Y%m%d-%H%M)"
   
   # Export logs
   tar -czf /tmp/logs-$(date +%Y%m%d).tar.gz /var/log/
   

3. Check AIDE:

   aide --check > /tmp/aide-report.txt
   

4. Review Audit Logs:

   ausearch -ts today > /tmp/audit-today.txt
   aureport --summary
   

5. Check Auth Logs:

   grep -E "(Failed|Accepted)" /var/log/auth.log | tail -100
   

Key Rotation

1Password Secrets:

1. Update secret in 1Password vault
2. Restart services: ./scripts/prod-start.sh restart

LUKS Key:

1. Generate new key
2. Add to LUKS: cryptsetup luksAddKey
3. Update 1Password
4. Remove old key: cryptsetup luksRemoveKey

JWT Keys:

1. Generate new keypair
2. Update in 1Password (JWT_PRIVATE_KEY, JWT_PUBLIC_KEY)
3. Restart services

7. Compliance Checklist

SOC 2 Type II

Security Maintenance

Quick Reference

Important Paths

Emergency Contacts

Built with Epic Flowstate