Environment: Web Universe Main Dev on web-03
"{\"env\": \"dev\", \"zone\": \"universe\", \"branch\": \"main\", \"db_app\": \"postgresql\", \"service\": \"web\", \"db_enabled\": true, \"pg_standby\": 0, \"pg_workers\": 1, \"description\": \"\", \"iam_enabled\": false, \"worker_1_ip\": \"\", \"eventbus_app\": \"kafka\", \"worker_1_fqdn\": \"db-web-universe-main-dev-postgresql-worker-01.fastorder.com\", \"search_enabled\": false, \"eventbus_enabled\": true, \"postgresql_enabled\": true, \"postgresql_run_verification\": true}"
This job has been restarted. You are viewing an older attempt. The logs and status shown below are from the latest retry.
This job completed successfully. You can review the steps or restart specific ones if needed.
⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...[0;34m[INFO][0m Using database engine from DB_ENGINE environment variable: postgresql
[0;34m[INFO][0m Cleaning up any existing locks...
[0;32m[1mStarting database engine: postgresql[0m
[1;33m═══════════════════════════════════════════════[0m
[0;34m[INFO][0m Using environment from web interface: web-universe-main-dev
[0;32m[2026-02-05 07:17:16][0m Using web-provided environment: web-universe-main-dev
[0;32m[2026-02-05 07:17:16][0m Service: web, Zone: universe, Branch: main, Env: dev
[0;32m✓[0m Environment initialized successfully (mode: general)
[0;34m[INFO][0m Checking observability cell readiness: obs-web-universe-main-dev
[1;32m[OK][0m Observability cell endpoints registered for web-universe-main-dev
[0;34m[INFO][0m Observability cell verified for web-universe-main-dev
[0;34m[INFO][0m Monitoring will be configured after PostgreSQL deployment (step 10-monitoring-setup.sh)
[0;34m[INFO][0m Citus mode ENABLED
[0;34m[INFO][0m → Coordinator + 1 worker(s) + 0 standby node(s) per worker
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up coordinator (Citus control plane)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-02-05 07:17:17 UTC] USER=unknown EUID=33 PID=148865 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 07:17:17 UTC] USER=unknown EUID=33 PID=148872 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 07:17:17 UTC] USER=unknown EUID=33 PID=148885 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-02-05 07:17:17 UTC] USER=unknown EUID=33 PID=148893 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 07:17:17 UTC] USER=unknown EUID=33 PID=148903 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-02-05 07:17:17 UTC] USER=unknown EUID=33 PID=148911 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for web in universe-dev...
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=ed3a3df2-33b1-4c79-b551-1f69750cf7b1 JOB_UUID=2af23dfc-1e8c-44dd-8cd0-595eeec92286
[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=ed3a3df2-33b1-4c79-b551-1f69750cf7b1)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.54
[0;34m[INFO][0m Primary hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[0;34m[INFO][0m Adding /etc/hosts entries for coordinator...
[0;34m[INFO][0m 1. db-web-universe-main-dev-postgresql.fastorder.com → 10.100.1.54 (primary/short)
[0;34m[INFO][0m 2. db-web-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.54 (compatibility)
[0;34m[INFO][0m ✅ db-web-universe-main-dev-postgresql.fastorder.com already exists with correct IP
[0;34m[INFO][0m ✅ db-web-universe-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
10.100.1.54 db-web-universe-main-dev-postgresql.fastorder.com
10.100.1.54 db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=ed3a3df2-33b1-4c79-b551-1f69750cf7b1)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe (FastOrder Universe)
Branch: main
Env: dev
Node: coordinator
Primary CN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Alt CN: web-universe-main-dev.fastorder.com
Internal IP: 10.100.1.54 (from topology: db-coordinator-postgresql)
Shared IP: 142.93.238.16 (legacy/public, also included in SANs)
Coordinator variants:
- db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
- db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 07:17:21 UTC] USER=www-data EUID=0 PID=149116 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator and /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:17:21 UTC] USER=www-data EUID=0 PID=149125 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-02-05 07:17:21 UTC] USER=www-data EUID=0 PID=149135 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-149060
[2026-02-05 07:17:21 UTC] USER=www-data EUID=0 PID=149144 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-149060/ra_root.crt
[2026-02-05 07:17:21 UTC] USER=www-data EUID=0 PID=149153 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-149060/ra_root.key
[2026-02-05 07:17:21 UTC] USER=www-data EUID=0 PID=149162 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-149060/ra_root.crt
[2026-02-05 07:17:21 UTC] USER=www-data EUID=0 PID=149171 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-149060/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[2026-02-05 07:17:24 UTC] USER=www-data EUID=0 PID=149307 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-149060/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 07:17:24 UTC] USER=www-data EUID=0 PID=149316 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-149060/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 07:17:24 UTC] USER=www-data EUID=0 PID=149326 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-02-05 07:17:24 UTC] USER=www-data EUID=0 PID=149335 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-149060/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:17:24 UTC] USER=www-data EUID=0 PID=149346 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 07:17:24 UTC] USER=www-data EUID=0 PID=149387 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 07:17:24 UTC] USER=www-data EUID=0 PID=149396 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 07:17:24 UTC] USER=www-data EUID=0 PID=149407 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 07:17:24 UTC] USER=www-data EUID=0 PID=149418 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 07:17:24 UTC] USER=www-data EUID=0 PID=149436 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔍 Verifying certificate...
Certificate details:
Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
--
X509v3 Subject Alternative Name:
DNS:db-web-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:db-web-universe-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-web-universe-main-dev-postgresql.fastorder.com, IP Address:10.100.1.54, IP Address:127.0.0.1, IP Address:142.93.238.16
X509v3 Subject Key Identifier:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Node: coordinator
Primary CN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-coordinator.service
3. Test SSL connection:
psql "host=db-web-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: postgres
Identifier: coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: coordinator
User (CN): postgres
Hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:17:25 UTC] USER=www-data EUID=0 PID=149499 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-02-05 07:17:25 UTC] USER=www-data EUID=0 PID=149510 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 07:17:25 UTC] USER=www-data EUID=0 PID=149519 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-02-05 07:17:25 UTC] USER=www-data EUID=0 PID=149528 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 07:17:25 UTC] USER=www-data EUID=0 PID=149539 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:17:25 UTC] USER=www-data EUID=0 PID=149556 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:17:25 UTC] USER=www-data EUID=0 PID=149566 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:17:25 UTC] USER=www-data EUID=0 PID=149575 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:17:25 UTC] USER=www-data EUID=0 PID=149584 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149593 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149602 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149620 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149629 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149646 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149655 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149664 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149673 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149684 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149694 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149705 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149714 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149732 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149770 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149779 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149788 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149806 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149828 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:17:26 UTC] USER=www-data EUID=0 PID=149856 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=149868 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=149879 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=149889 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=149898 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=149909 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=149920 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=149930 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=149940 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=149949 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=149958 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=149989 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=149998 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=150010 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=150019 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=150028 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=150042 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=150052 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=150062 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=150071 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:17:27 UTC] USER=www-data EUID=0 PID=150080 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:17:28 UTC] USER=www-data EUID=0 PID=150089 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:28 UTC] USER=www-data EUID=0 PID=150109 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:17:28 UTC] USER=www-data EUID=0 PID=150118 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:17:28 UTC] USER=www-data EUID=0 PID=150127 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:17:28 UTC] USER=www-data EUID=0 PID=150157 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:17:28 UTC] USER=www-data EUID=0 PID=150166 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:17:28 UTC] USER=www-data EUID=0 PID=150178 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:17:28 UTC] USER=www-data EUID=0 PID=150190 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:28 UTC] USER=www-data EUID=0 PID=150213 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:17:28 UTC] USER=www-data EUID=0 PID=150223 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:17:28 UTC] USER=www-data EUID=0 PID=150239 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:28 UTC] USER=www-data EUID=0 PID=150266 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:17:29 UTC] USER=www-data EUID=0 PID=150288 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:17:29 UTC] USER=www-data EUID=0 PID=150318 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:17:29 UTC] USER=www-data EUID=0 PID=150327 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:17:29 UTC] USER=www-data EUID=0 PID=150340 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:17:29 UTC] USER=www-data EUID=0 PID=150353 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:17:29 UTC] USER=www-data EUID=0 PID=150372 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:17:29 UTC] USER=www-data EUID=0 PID=150384 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:17:29 UTC] USER=www-data EUID=0 PID=150398 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 07:17:29 UTC] USER=www-data EUID=0 PID=150408 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 07:17:29 UTC] USER=www-data EUID=0 PID=150419 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 07:17:29 UTC] USER=www-data EUID=0 PID=150429 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:17:29 UTC] USER=www-data EUID=0 PID=150439 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:17:29 UTC] USER=www-data EUID=0 PID=150449 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:17:29 UTC] USER=www-data EUID=0 PID=150458 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: postgres
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: postgres
Identifier: coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: coordinator
User (CN): postgres
Hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:17:30 UTC] USER=www-data EUID=0 PID=150542 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-02-05 07:17:30 UTC] USER=www-data EUID=0 PID=150578 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:17:30 UTC] USER=www-data EUID=0 PID=150605 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:17:30 UTC] USER=www-data EUID=0 PID=150614 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:17:30 UTC] USER=www-data EUID=0 PID=150623 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:17:30 UTC] USER=www-data EUID=0 PID=150641 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:17:30 UTC] USER=www-data EUID=0 PID=150652 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:17:30 UTC] USER=www-data EUID=0 PID=150661 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:17:30 UTC] USER=www-data EUID=0 PID=150675 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:17:30 UTC] USER=www-data EUID=0 PID=150688 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:17:30 UTC] USER=www-data EUID=0 PID=150697 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:17:30 UTC] USER=www-data EUID=0 PID=150715 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:17:30 UTC] USER=www-data EUID=0 PID=150726 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:17:30 UTC] USER=www-data EUID=0 PID=150735 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:17:30 UTC] USER=www-data EUID=0 PID=150744 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150756 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150765 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150774 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150783 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150792 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150818 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150827 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150836 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150845 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150854 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150864 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150880 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150898 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150916 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150925 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150935 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150945 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150956 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150965 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150974 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=150983 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=151012 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=151021 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:17:31 UTC] USER=www-data EUID=0 PID=151030 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151039 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151052 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151063 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151073 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151082 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151091 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151100 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151109 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151118 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151127 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151136 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151145 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151154 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151165 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151176 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151186 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151196 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151205 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151214 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151223 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151232 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151241 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151250 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151259 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151268 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151277 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151286 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151295 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151306 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151316 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151325 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151334 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151343 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151352 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 07:17:32 UTC] USER=www-data EUID=0 PID=151361 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:17:33 UTC] USER=www-data EUID=0 PID=151370 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:17:33 UTC] USER=www-data EUID=0 PID=151379 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:17:33 UTC] USER=www-data EUID=0 PID=151388 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: postgres
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres
[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=ed3a3df2-33b1-4c79-b551-1f69750cf7b1)
[INFO] 📦 02 setup pg instance...
[0;34m[DEADLOCK-PREVENTION][0m Deadlock prevention library loaded
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Using existing db-coordinator-postgresql environment: db-web-universe-main-dev-postgresql-coordinator.fastorder.com (10.100.1.54)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.54
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m Data dir: /data/postgresql/17/web-universe-main-dev/coordinator
[0;34m[INFO][0m Port: 5432
[0;34m[INFO][0m Hostname: db-web-universe-main-dev-postgresql-coordinator
[2026-02-05 07:17:34 UTC] USER=www-data EUID=0 PID=151539 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:17:34 UTC] USER=www-data EUID=0 PID=151605 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[1;33m[WARN][0m Server certificate not found at /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[0;34m[INFO][0m Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe (FastOrder Universe)
Branch: main
Env: dev
Node: coordinator
Primary CN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Alt CN: web-universe-main-dev.fastorder.com
Internal IP: 10.100.1.54 (from topology: db-coordinator-postgresql)
Shared IP: 142.93.238.16 (legacy/public, also included in SANs)
Coordinator variants:
- db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
- db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 07:17:35 UTC] USER=www-data EUID=0 PID=151647 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator and /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:17:35 UTC] USER=www-data EUID=0 PID=151658 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-02-05 07:17:35 UTC] USER=www-data EUID=0 PID=151673 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-151612
[2026-02-05 07:17:35 UTC] USER=www-data EUID=0 PID=151684 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-151612/ra_root.crt
[2026-02-05 07:17:35 UTC] USER=www-data EUID=0 PID=151703 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-151612/ra_root.crt
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[2026-02-05 07:17:37 UTC] USER=www-data EUID=0 PID=151792 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-02-05 07:17:37 UTC] USER=www-data EUID=0 PID=151801 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-151612/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:17:37 UTC] USER=www-data EUID=0 PID=151810 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:17:37 UTC] USER=www-data EUID=0 PID=151819 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:17:37 UTC] USER=www-data EUID=0 PID=151828 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 07:17:37 UTC] USER=www-data EUID=0 PID=151840 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 07:17:37 UTC] USER=www-data EUID=0 PID=151850 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 07:17:37 UTC] USER=www-data EUID=0 PID=151859 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 07:17:37 UTC] USER=www-data EUID=0 PID=151877 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔍 Verifying certificate...
Certificate details:
Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
--
X509v3 Subject Alternative Name:
DNS:db-web-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:db-web-universe-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-web-universe-main-dev-postgresql.fastorder.com, IP Address:10.100.1.54, IP Address:127.0.0.1, IP Address:142.93.238.16
X509v3 Subject Key Identifier:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Node: coordinator
Primary CN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-coordinator.service
3. Test SSL connection:
psql "host=db-web-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-02-05 07:17:37 UTC] USER=www-data EUID=0 PID=151922 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 07:17:37 UTC] USER=www-data EUID=0 PID=151932 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 07:17:37 UTC] USER=www-data EUID=0 PID=151942 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[0;32m[OK][0m mTLS certificates OK (server cert + client certs verified) and keys secured
[0;34m[INFO][0m Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-02-05 07:17:37 UTC] USER=www-data EUID=0 PID=151968 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 07:17:38 UTC] USER=www-data EUID=0 PID=151998 ACTION=passthru ARGS=systemctl stop postgresql
[1;33m[WARN][0m Cleaning stale socket directory /var/run/postgresql-web-universe-main-dev-coordinator
[2026-02-05 07:17:38 UTC] USER=www-data EUID=0 PID=152044 ACTION=fsop ARGS=rm -rf /var/run/postgresql-web-universe-main-dev-coordinator
[0;32m[OK][0m No conflicting Postgres left on port 5432
[0;32m[OK][0m Using postgres password from vault provider
[2026-02-05 07:17:40 UTC] USER=www-data EUID=0 PID=152303 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.qqbmBZ
[2026-02-05 07:17:40 UTC] USER=www-data EUID=0 PID=152327 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.qqbmBZ
[2026-02-05 07:17:40 UTC] USER=www-data EUID=0 PID=152353 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev
[2026-02-05 07:17:40 UTC] USER=www-data EUID=0 PID=152390 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev
[2026-02-05 07:17:41 UTC] USER=www-data EUID=0 PID=152413 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/web-universe-main-dev
[0;34m[INFO][0m Initializing cluster in /data/postgresql/17/web-universe-main-dev/coordinator (SCRAM; pwfile)
[1;33m[WARN][0m Removing existing data directory: /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 07:17:41 UTC] USER=www-data EUID=0 PID=152446 ACTION=fsop ARGS=rm -rf /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 07:17:41 UTC] USER=www-data EUID=0 PID=152473 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 07:17:41 UTC] USER=www-data EUID=0 PID=152495 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 07:17:42 UTC] USER=www-data EUID=0 PID=152519 ACTION=fsop ARGS=chmod 700 /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 07:17:42 UTC] USER=www-data EUID=0 PID=152586 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-web-universe-main-dev-coordinator
[2026-02-05 07:17:42 UTC] USER=www-data EUID=0 PID=152595 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/web-universe-main-dev/coordinator --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.qqbmBZ
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /data/postgresql/17/web-universe-main-dev/coordinator ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
/usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/web-universe-main-dev/coordinator -l logfile start
[0;32m[OK][0m initdb complete
[2026-02-05 07:17:44 UTC] USER=www-data EUID=0 PID=152667 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.qqbmBZ
[0;34m[INFO][0m Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[0;32m[OK][0m postgresql.conf updated successfully
[0;34m[INFO][0m Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-02-05 07:17:44 UTC] USER=www-data EUID=0 PID=152716 ACTION=fsop ARGS=cp /tmp/tmp.hnoRH5Tdqo /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
[2026-02-05 07:17:44 UTC] USER=www-data EUID=0 PID=152738 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
[0;32m[OK][0m pg_hba.conf updated
[0;34m[INFO][0m Creating systemd unit: /etc/systemd/system/postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 07:17:44 UTC] USER=www-data EUID=0 PID=152808 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.Rzyk4z /etc/systemd/system/postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m systemd unit written
[2026-02-05 07:17:45 UTC] USER=www-data EUID=0 PID=152868 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 07:17:45 UTC] USER=www-data EUID=0 PID=152893 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 07:17:45 UTC] USER=www-data EUID=0 PID=152914 ACTION=passthru ARGS=systemctl daemon-reload
[0;34m[INFO][0m Starting PostgreSQL instance...
[2026-02-05 07:17:46 UTC] USER=www-data EUID=0 PID=153054 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[0;34m[INFO][0m Waiting for ACTIVE (systemd)…
[2026-02-05 07:17:47 UTC] USER=www-data EUID=0 PID=153125 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m Service ACTIVE
[0;34m[INFO][0m Waiting for port 5432 bind…
[0;32m[OK][0m Port bound
[0;34m[INFO][0m Waiting pg_isready (socket)…
[0;32m[OK][0m Readiness via socket OK
[0;34m[INFO][0m Waiting pg_isready (TCP db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432)…
[0;32m[OK][0m Startup sequence complete
[0;34m[INFO][0m Validating core security GUCs (via local socket)…
[0;32m[OK][0m Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[0;34m[INFO][0m Provisioning application database and Debezium role (if not exists)...
[0;34m[INFO][0m Checking if database fastorder_web_universe_main_dev_db exists...
[0;34m[INFO][0m DB check result: exit_code=0, output='[2026-02-05 07:17:48 UTC] USER=www-data EUID=0 PID=153316 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_web_universe_main_dev_db''
[0;34m[INFO][0m Creating database fastorder_web_universe_main_dev_db...
[2026-02-05 07:17:48 UTC] USER=www-data EUID=0 PID=153343 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_web_universe_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[0;32m[OK][0m Database fastorder_web_universe_main_dev_db created
[0;34m[INFO][0m Checking if role debezium_user exists...
[0;34m[INFO][0m Role check result: exit_code=0, output='[2026-02-05 07:17:48 UTC] USER=www-data EUID=0 PID=153380 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[0;34m[INFO][0m Creating role debezium_user...
[2026-02-05 07:17:48 UTC] USER=www-data EUID=0 PID=153409 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'tjPxFSuckpOMTnI0CyoF2pZV';
CREATE ROLE
[0;32m[OK][0m Role debezium_user created
[2026-02-05 07:17:48 UTC] USER=www-data EUID=0 PID=153436 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_web_universe_main_dev_db" TO debezium_user;
GRANT
[0;32m[OK][0m Application DB (fastorder_web_universe_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[0;34m[INFO][0m Applying connection and memory optimizations...
[0;34m[INFO][0m Current settings: max_connections=100, work_mem=4MB
[0;34m[INFO][0m Target settings (coordinator): max_connections=150, work_mem=8MB
[2026-02-05 07:17:49 UTC] USER=www-data EUID=0 PID=153515 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 150;
ALTER SYSTEM
[2026-02-05 07:17:49 UTC] USER=www-data EUID=0 PID=153543 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-02-05 07:17:49 UTC] USER=www-data EUID=0 PID=153598 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
pg_reload_conf
----------------
t
(1 row)
[0;32m[OK][0m Settings applied to postgresql.auto.conf
[2026-02-05 07:17:49 UTC] USER=www-data EUID=0 PID=153614 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/coordinator/standby.signal
[0;34m[INFO][0m Service recently started (2s ago) - restarting to apply max_connections...
[0;34m[INFO][0m Stopping service...
[2026-02-05 07:17:49 UTC] USER=www-data EUID=0 PID=153636 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[0;34m[INFO][0m Waiting for port 5432 to be released...
[0;32m[OK][0m Port 5432 released
[0;34m[INFO][0m Starting service...
[2026-02-05 07:17:53 UTC] USER=www-data EUID=0 PID=153730 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 07:17:58 UTC] USER=www-data EUID=0 PID=153942 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m ✅ Optimization complete: max_connections=150, work_mem=8MB
[0;34m[INFO][0m Setting postgres password via centralized script... for coordinator
[0;34m[INFO][0m Temporarily disabling synchronous_commit on coordinator for password setting...
[0;32m[OK][0m Disabled synchronous_commit (was: on)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: coordinator[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-coordinator[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mInitial setup: Using password from initdb[0m
[0;32m✓ PostgreSQL password already set during initdb[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/coordinator
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/coordinator[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials coordinator[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
[0;34m[INFO][0m Restoring synchronous_commit on coordinator...
[0;32m[OK][0m Restored synchronous_commit to: on
[0;32m[OK][0m Password set and persisted
[0;34m[INFO][0m Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.54
[0;34m[INFO][0m Primary hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[0;34m[INFO][0m Adding /etc/hosts entries for coordinator...
[0;34m[INFO][0m 1. db-web-universe-main-dev-postgresql.fastorder.com → 10.100.1.54 (primary/short)
[0;34m[INFO][0m 2. db-web-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.54 (compatibility)
[0;34m[INFO][0m ✅ db-web-universe-main-dev-postgresql.fastorder.com already exists with correct IP
[0;34m[INFO][0m ✅ db-web-universe-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
10.100.1.54 db-web-universe-main-dev-postgresql.fastorder.com
10.100.1.54 db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[0;32m[OK][0m PostgreSQL 'web-universe-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt \
sslcert=/home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt \
sslkey=/home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key \
host=db-web-universe-main-dev-postgresql-coordinator port=5432 dbname=postgres user=postgres"
File been compeleted perfectly: 02-setup-pg-instance
[0;34m[INFO][0m Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: PostgreSQL
[INFO] Identifier: web-universe-main-dev-postgresql-coordinator
[INFO] Identifier Parent: coordinator
[INFO] IP: 10.100.1.54
[INFO] Port: 5432
[INFO] FQDN: db-web-universe-main-dev-postgresql-coordinator
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 90feffd7-89fb-4afb-a63f-cc975d7e928c
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[0;32m[OK][0m PostgreSQL node registered to observability API
[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=ed3a3df2-33b1-4c79-b551-1f69750cf7b1)
[INFO] 📦 03 role...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 07:18:10 UTC] USER=www-data EUID=0 PID=154778 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/coordinator/standby.signal
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: debezium_user
Identifier: coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: coordinator
User (CN): debezium_user
Hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:18:11 UTC] USER=www-data EUID=0 PID=154992 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-debezium_user
[2026-02-05 07:18:11 UTC] USER=www-data EUID=0 PID=155010 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-debezium_user/ra_root.key
[2026-02-05 07:18:11 UTC] USER=www-data EUID=0 PID=155020 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.crt
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:18:12 UTC] USER=www-data EUID=0 PID=155053 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:18:12 UTC] USER=www-data EUID=0 PID=155062 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:18:12 UTC] USER=www-data EUID=0 PID=155071 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 07:18:12 UTC] USER=www-data EUID=0 PID=155080 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 07:18:12 UTC] USER=www-data EUID=0 PID=155089 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:18:12 UTC] USER=www-data EUID=0 PID=155098 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:18:12 UTC] USER=www-data EUID=0 PID=155107 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 07:18:12 UTC] USER=www-data EUID=0 PID=155116 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 07:18:12 UTC] USER=www-data EUID=0 PID=155125 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 07:18:12 UTC] USER=www-data EUID=0 PID=155145 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:18:12 UTC] USER=www-data EUID=0 PID=155164 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:18:12 UTC] USER=www-data EUID=0 PID=155173 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 07:18:13 UTC] USER=www-data EUID=0 PID=155191 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 07:18:13 UTC] USER=www-data EUID=0 PID=155209 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:18:13 UTC] USER=www-data EUID=0 PID=155218 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:13 UTC] USER=www-data EUID=0 PID=155248 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:13 UTC] USER=www-data EUID=0 PID=155258 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:13 UTC] USER=www-data EUID=0 PID=155278 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:13 UTC] USER=www-data EUID=0 PID=155287 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:13 UTC] USER=www-data EUID=0 PID=155296 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 07:18:13 UTC] USER=www-data EUID=0 PID=155305 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 07:18:13 UTC] USER=www-data EUID=0 PID=155314 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:18:13 UTC] USER=www-data EUID=0 PID=155323 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:18:13 UTC] USER=www-data EUID=0 PID=155332 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 07:18:13 UTC] USER=www-data EUID=0 PID=155344 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 07:18:13 UTC] USER=www-data EUID=0 PID=155353 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 07:18:14 UTC] USER=www-data EUID=0 PID=155363 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:14 UTC] USER=www-data EUID=0 PID=155383 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:14 UTC] USER=www-data EUID=0 PID=155395 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:14 UTC] USER=www-data EUID=0 PID=155404 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:14 UTC] USER=www-data EUID=0 PID=155413 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:14 UTC] USER=www-data EUID=0 PID=155422 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:14 UTC] USER=www-data EUID=0 PID=155432 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 07:18:14 UTC] USER=www-data EUID=0 PID=155443 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 07:18:14 UTC] USER=www-data EUID=0 PID=155467 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:18:14 UTC] USER=www-data EUID=0 PID=155486 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 07:18:14 UTC] USER=www-data EUID=0 PID=155495 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 07:18:14 UTC] USER=www-data EUID=0 PID=155506 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 07:18:14 UTC] USER=www-data EUID=0 PID=155517 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155553 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155562 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155571 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155580 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155589 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155617 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155626 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155635 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155647 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155662 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155672 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155682 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155693 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155704 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155714 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155724 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155733 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155742 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155751 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155760 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155771 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155780 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155789 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155798 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155819 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155830 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155841 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 07:18:15 UTC] USER=www-data EUID=0 PID=155853 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
✅ Symlinked client-key.pem
[2026-02-05 07:18:16 UTC] USER=www-data EUID=0 PID=155876 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:18:16 UTC] USER=www-data EUID=0 PID=155885 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:18:16 UTC] USER=www-data EUID=0 PID=155897 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: debezium_user
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U debezium_user -d postgres
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: coordinator[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-coordinator[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mGenerating new secure password...[0m
[0;34mUser fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: fastorder_admin_gd[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials coordinator[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: coordinator/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-web-universe-main-dev-postgresql.fastorder.com
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: fastorder_admin_gd
Identifier: coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: coordinator
User (CN): fastorder_admin_gd
Hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:18:25 UTC] USER=www-data EUID=0 PID=156401 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-fastorder_admin_gd
[2026-02-05 07:18:25 UTC] USER=www-data EUID=0 PID=156429 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-02-05 07:18:25 UTC] USER=www-data EUID=0 PID=156438 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:18:25 UTC] USER=www-data EUID=0 PID=156505 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:18:25 UTC] USER=www-data EUID=0 PID=156518 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:18:25 UTC] USER=www-data EUID=0 PID=156535 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 07:18:25 UTC] USER=www-data EUID=0 PID=156554 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 07:18:25 UTC] USER=www-data EUID=0 PID=156564 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:18:25 UTC] USER=www-data EUID=0 PID=156586 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:18:25 UTC] USER=www-data EUID=0 PID=156595 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 07:18:25 UTC] USER=www-data EUID=0 PID=156604 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 07:18:26 UTC] USER=www-data EUID=0 PID=156613 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 07:18:26 UTC] USER=www-data EUID=0 PID=156622 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:18:26 UTC] USER=www-data EUID=0 PID=156631 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 07:18:26 UTC] USER=www-data EUID=0 PID=156640 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 07:18:26 UTC] USER=www-data EUID=0 PID=156652 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:18:26 UTC] USER=www-data EUID=0 PID=156661 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:18:26 UTC] USER=www-data EUID=0 PID=156674 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 07:18:26 UTC] USER=www-data EUID=0 PID=156684 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:18:26 UTC] USER=www-data EUID=0 PID=156693 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 07:18:26 UTC] USER=www-data EUID=0 PID=156716 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:18:26 UTC] USER=www-data EUID=0 PID=156725 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:26 UTC] USER=www-data EUID=0 PID=156763 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:26 UTC] USER=www-data EUID=0 PID=156781 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:26 UTC] USER=www-data EUID=0 PID=156792 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:26 UTC] USER=www-data EUID=0 PID=156852 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:18:27 UTC] USER=www-data EUID=0 PID=156871 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:18:27 UTC] USER=www-data EUID=0 PID=156880 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 07:18:27 UTC] USER=www-data EUID=0 PID=156899 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:27 UTC] USER=www-data EUID=0 PID=156911 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:27 UTC] USER=www-data EUID=0 PID=156920 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:27 UTC] USER=www-data EUID=0 PID=156947 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:27 UTC] USER=www-data EUID=0 PID=156958 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:27 UTC] USER=www-data EUID=0 PID=156975 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 07:18:27 UTC] USER=www-data EUID=0 PID=156984 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 07:18:27 UTC] USER=www-data EUID=0 PID=156993 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:18:27 UTC] USER=www-data EUID=0 PID=157004 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:18:27 UTC] USER=www-data EUID=0 PID=157013 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:18:28 UTC] USER=www-data EUID=0 PID=157024 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 07:18:28 UTC] USER=www-data EUID=0 PID=157037 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 07:18:28 UTC] USER=www-data EUID=0 PID=157047 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:28 UTC] USER=www-data EUID=0 PID=157057 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:28 UTC] USER=www-data EUID=0 PID=157071 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:28 UTC] USER=www-data EUID=0 PID=157080 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:28 UTC] USER=www-data EUID=0 PID=157089 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:28 UTC] USER=www-data EUID=0 PID=157098 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:28 UTC] USER=www-data EUID=0 PID=157108 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 07:18:28 UTC] USER=www-data EUID=0 PID=157125 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 07:18:28 UTC] USER=www-data EUID=0 PID=157135 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:18:28 UTC] USER=www-data EUID=0 PID=157145 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:18:28 UTC] USER=www-data EUID=0 PID=157154 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:18:28 UTC] USER=www-data EUID=0 PID=157164 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 07:18:28 UTC] USER=www-data EUID=0 PID=157188 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:28 UTC] USER=www-data EUID=0 PID=157200 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:29 UTC] USER=www-data EUID=0 PID=157210 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:29 UTC] USER=www-data EUID=0 PID=157219 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:29 UTC] USER=www-data EUID=0 PID=157228 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:29 UTC] USER=www-data EUID=0 PID=157237 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:18:29 UTC] USER=www-data EUID=0 PID=157255 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 07:18:29 UTC] USER=www-data EUID=0 PID=157265 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:18:29 UTC] USER=www-data EUID=0 PID=157274 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:18:29 UTC] USER=www-data EUID=0 PID=157290 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:18:29 UTC] USER=www-data EUID=0 PID=157311 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 07:18:29 UTC] USER=www-data EUID=0 PID=157321 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:18:29 UTC] USER=www-data EUID=0 PID=157332 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:18:29 UTC] USER=www-data EUID=0 PID=157345 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:18:29 UTC] USER=www-data EUID=0 PID=157354 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 07:18:29 UTC] USER=www-data EUID=0 PID=157363 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 07:18:29 UTC] USER=www-data EUID=0 PID=157372 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 07:18:29 UTC] USER=www-data EUID=0 PID=157381 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:18:30 UTC] USER=www-data EUID=0 PID=157390 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:18:30 UTC] USER=www-data EUID=0 PID=157399 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:18:30 UTC] USER=www-data EUID=0 PID=157408 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: fastorder_admin_gd
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U fastorder_admin_gd -d postgres
🧱 Connecting via Unix socket to create role and database...
Socket: /var/run/postgresql-web-universe-main-dev-coordinator:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️ Database fastorder_web_universe_main_dev_db already exists, skipping creation
[2026-02-05 07:18:30 UTC] USER=www-data EUID=0 PID=157499 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️ Using pg_hba.conf: /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-02-05 07:18:30 UTC] USER=www-data EUID=0 PID=157548 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[0;34m=== Pre-flight Checks ===[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;32m✓[0m AWS Secrets Manager accessible
[0;34m=== Retrieving Credentials from AWS ===[0m
ℹ️ Retrieving PostgreSQL credentials for: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️ Fetching secret: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials loaded for coordinator/fastorder_admin_gd: fastorder_admin_gd@db-web-universe-main-dev-postgresql.fastorder.com:5432/fastorder_web_universe_main_dev_db
[0;32m✓[0m Credentials retrieved: fastorder_admin_gd@db-web-universe-main-dev-postgresql.fastorder.com:5432/fastorder_web_universe_main_dev_db
[0;34m╔════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Test Suite (AWS Secrets MGR) ║[0m
[0;34m╚════════════════════════════════════════════╝[0m
[0;34m=== PostgreSQL Authentication Test ===[0m
[0;31m✗[0m PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-web-universe-main-dev-postgresql.fastorder.com" (10.100.1.54), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 07:18:36 UTC] USER=www-data EUID=0 PID=157853 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/coordinator/standby.signal
── fast setup ─────────────────────────────────────────────
NAME : web-universe-main-dev
IDENTIFIER : coordinator
PG HOST : db-web-universe-main-dev-postgresql.fastorder.com:5432
ROLE : debezium_user
DB : fastorder_web_universe_main_dev_db
SCHEMA : web
AUTH MODE : scram (scram=password over TLS | cert=mTLS)
SUBNET ALLOW: 10.201.0.0/16
CONNECT /32 : 142.93.238.16
SSL DIR : /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
DNS → 10.100.1.54
CA : /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: coordinator[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-coordinator[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mGenerating new secure password...[0m
[0;34mUser debezium_user does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: debezium_user[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials coordinator[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: coordinator/debezium_user
✓ Retrieved password from secrets vault
password : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️ Role debezium_user exists, updating
[2026-02-05 07:18:43 UTC] USER=www-data EUID=0 PID=158440 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️ Database fastorder_web_universe_main_dev_db already exists
[2026-02-05 07:18:44 UTC] USER=www-data EUID=0 PID=158472 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d fastorder_web_universe_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️ Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=ed3a3df2-33b1-4c79-b551-1f69750cf7b1)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
ℹ️ Service-specific setup (web) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)
🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=web
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
🔍 DEBUG_CHECKPOINT_03: Found service-specific run.sh for: web
[DEBUG] Tracking substep start: steps/01-install/steps/web (RUN_UUID=ed3a3df2-33b1-4c79-b551-1f69750cf7b1)
[INFO] 🔸 Service: web (service-specific)
🔍 DEBUG_CHECKPOINT_04: Executing service-specific: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Web Service Schema Provisioning
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service: web
[INFO] Zone: universe
[INFO] Branch: main
[INFO] Environment: dev
[INFO] Identifier: coordinator
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] 📦 Running: 01-init-schema.sh...
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Initializing config schema
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Identifier: coordinator
Database: fastorder_web_universe_main_dev_db
Host: db-web-universe-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking if config schema exists...
[INFO] Creating config schema and tables...
CREATE EXTENSION
CREATE SCHEMA
CREATE TABLE
COMMENT
CREATE TABLE
COMMENT
CREATE TABLE
COMMENT
CREATE INDEX
CREATE INDEX
CREATE INDEX
INSERT 0 1
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
NOTICE: trigger "trg_public_defaults_version" for relation "config.public_defaults" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
NOTICE: trigger "trg_feature_flags_version" for relation "config.feature_flags" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
NOTICE: trigger "trg_public_defaults_set_updated_at" for relation "config.public_defaults" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
NOTICE: trigger "trg_feature_flags_set_updated_at" for relation "config.feature_flags" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
[OK] Config schema and tables created
[INFO] Seeding initial guest services data...
INSERT 0 9
INSERT 0 1
[OK] Initial data seeded
[INFO] Verifying config schema...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Config Schema Verification
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Guest services: 9
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] Config schema initialization complete
[OK] ✓ 01-init-schema.sh completed
[INFO] ═══════════════════════════════════════════════════════════════
[OK] Web service schema provisioning complete
[INFO] ═══════════════════════════════════════════════════════════════
[0;32m✓[0m ✅ Coordinator setup completed
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up 1 worker(s) (Citus data nodes)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up worker: worker-01
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-02-05 07:18:48 UTC] USER=unknown EUID=33 PID=158736 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 07:18:48 UTC] USER=unknown EUID=33 PID=158743 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 07:18:48 UTC] USER=unknown EUID=33 PID=158752 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-02-05 07:18:48 UTC] USER=unknown EUID=33 PID=158773 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 07:18:48 UTC] USER=unknown EUID=33 PID=158785 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-02-05 07:18:48 UTC] USER=unknown EUID=33 PID=158801 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for web in universe-dev...
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=ed3a3df2-33b1-4c79-b551-1f69750cf7b1 JOB_UUID=2af23dfc-1e8c-44dd-8cd0-595eeec92286
[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=ed3a3df2-33b1-4c79-b551-1f69750cf7b1)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.243
[0;34m[INFO][0m Primary hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[0;34m[INFO][0m Adding /etc/hosts entry for worker-01...
[0;34m[INFO][0m db-web-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.243
[0;34m[INFO][0m ✅ db-web-universe-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
10.100.1.243 db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=ed3a3df2-33b1-4c79-b551-1f69750cf7b1)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe (FastOrder Universe)
Branch: main
Env: dev
Node: worker-01
Primary CN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Alt CN: web-universe-main-dev.fastorder.com
Internal IP: 10.100.1.243 (from topology: db-worker-01-postgresql)
Shared IP: 142.93.238.16 (legacy/public, also included in SANs)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 07:18:51 UTC] USER=www-data EUID=0 PID=159055 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01 and /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:18:51 UTC] USER=www-data EUID=0 PID=159070 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-02-05 07:18:51 UTC] USER=www-data EUID=0 PID=159081 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-158953
[2026-02-05 07:18:51 UTC] USER=www-data EUID=0 PID=159101 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-158953/ra_root.key
[2026-02-05 07:18:51 UTC] USER=www-data EUID=0 PID=159110 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-158953/ra_root.crt
[2026-02-05 07:18:51 UTC] USER=www-data EUID=0 PID=159119 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-158953/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[2026-02-05 07:18:54 UTC] USER=www-data EUID=0 PID=159208 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-158953/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 07:18:54 UTC] USER=www-data EUID=0 PID=159217 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-02-05 07:18:54 UTC] USER=www-data EUID=0 PID=159227 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-158953/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:18:54 UTC] USER=www-data EUID=0 PID=159236 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 07:18:54 UTC] USER=www-data EUID=0 PID=159265 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 07:18:54 UTC] USER=www-data EUID=0 PID=159274 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 07:18:54 UTC] USER=www-data EUID=0 PID=159283 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 07:18:55 UTC] USER=www-data EUID=0 PID=159303 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:18:55 UTC] USER=www-data EUID=0 PID=159312 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔍 Verifying certificate...
Certificate details:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Node: worker-01
Primary CN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-worker-01.service
3. Test SSL connection:
psql "host=db-web-universe-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: postgres
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: worker-01
User (CN): postgres
Hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:18:55 UTC] USER=www-data EUID=0 PID=159404 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-02-05 07:18:55 UTC] USER=www-data EUID=0 PID=159414 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 07:18:55 UTC] USER=www-data EUID=0 PID=159431 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-02-05 07:18:55 UTC] USER=www-data EUID=0 PID=159448 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 07:18:55 UTC] USER=www-data EUID=0 PID=159458 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:18:55 UTC] USER=www-data EUID=0 PID=159479 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:18:55 UTC] USER=www-data EUID=0 PID=159488 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:18:55 UTC] USER=www-data EUID=0 PID=159497 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159506 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159516 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159542 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159569 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159578 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159587 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159597 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159608 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159618 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159631 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159642 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159652 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159663 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159690 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159699 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159708 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159717 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159726 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159735 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159744 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159753 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:18:56 UTC] USER=www-data EUID=0 PID=159762 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:18:57 UTC] USER=www-data EUID=0 PID=159771 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:18:57 UTC] USER=www-data EUID=0 PID=159780 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:18:57 UTC] USER=www-data EUID=0 PID=159791 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:18:57 UTC] USER=www-data EUID=0 PID=159802 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:18:57 UTC] USER=www-data EUID=0 PID=159821 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:57 UTC] USER=www-data EUID=0 PID=159835 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:57 UTC] USER=www-data EUID=0 PID=159849 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:18:57 UTC] USER=www-data EUID=0 PID=159864 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:18:57 UTC] USER=www-data EUID=0 PID=159874 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:18:57 UTC] USER=www-data EUID=0 PID=159885 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:18:57 UTC] USER=www-data EUID=0 PID=159905 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:18:57 UTC] USER=www-data EUID=0 PID=159914 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:18:57 UTC] USER=www-data EUID=0 PID=159923 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:18:58 UTC] USER=www-data EUID=0 PID=159932 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:18:58 UTC] USER=www-data EUID=0 PID=159942 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:18:58 UTC] USER=www-data EUID=0 PID=159952 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:18:58 UTC] USER=www-data EUID=0 PID=159973 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:58 UTC] USER=www-data EUID=0 PID=159987 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:58 UTC] USER=www-data EUID=0 PID=159998 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:18:58 UTC] USER=www-data EUID=0 PID=160018 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:18:58 UTC] USER=www-data EUID=0 PID=160044 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:18:58 UTC] USER=www-data EUID=0 PID=160053 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:18:58 UTC] USER=www-data EUID=0 PID=160067 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:18:58 UTC] USER=www-data EUID=0 PID=160079 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:18:58 UTC] USER=www-data EUID=0 PID=160090 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:18:58 UTC] USER=www-data EUID=0 PID=160101 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:18:58 UTC] USER=www-data EUID=0 PID=160111 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:18:58 UTC] USER=www-data EUID=0 PID=160120 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:18:58 UTC] USER=www-data EUID=0 PID=160139 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:18:59 UTC] USER=www-data EUID=0 PID=160176 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:18:59 UTC] USER=www-data EUID=0 PID=160197 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:18:59 UTC] USER=www-data EUID=0 PID=160206 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:18:59 UTC] USER=www-data EUID=0 PID=160224 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:18:59 UTC] USER=www-data EUID=0 PID=160253 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
✅ Symlinked ca.pem
[2026-02-05 07:18:59 UTC] USER=www-data EUID=0 PID=160275 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 07:18:59 UTC] USER=www-data EUID=0 PID=160286 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 07:18:59 UTC] USER=www-data EUID=0 PID=160299 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:18:59 UTC] USER=www-data EUID=0 PID=160308 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:18:59 UTC] USER=www-data EUID=0 PID=160317 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: postgres
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: postgres
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: worker-01
User (CN): postgres
Hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:19:00 UTC] USER=www-data EUID=0 PID=160369 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-02-05 07:19:00 UTC] USER=www-data EUID=0 PID=160390 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-02-05 07:19:00 UTC] USER=www-data EUID=0 PID=160409 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:00 UTC] USER=www-data EUID=0 PID=160463 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:00 UTC] USER=www-data EUID=0 PID=160472 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:00 UTC] USER=www-data EUID=0 PID=160483 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:19:00 UTC] USER=www-data EUID=0 PID=160494 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:19:00 UTC] USER=www-data EUID=0 PID=160512 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160521 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160532 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160541 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160571 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160580 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160596 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160608 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160622 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160634 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160643 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160652 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160662 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160677 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160692 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160701 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160727 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160736 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160755 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160764 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160773 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160782 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160791 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:01 UTC] USER=www-data EUID=0 PID=160809 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=160818 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=160827 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=160837 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=160847 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=160856 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=160874 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=160904 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=160913 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=160922 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=160940 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=160951 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=160964 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=160974 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=160984 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=160993 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=161002 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=161013 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=161040 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=161049 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:02 UTC] USER=www-data EUID=0 PID=161058 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161067 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161076 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161085 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161095 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161105 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161114 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161123 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161132 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161141 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161150 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161159 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161168 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161177 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161186 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161195 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161204 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161215 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161237 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161246 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
✅ Symlinked client-cert.pem
✅ Symlinked client-key.pem
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161273 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161282 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161291 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:19:03 UTC] USER=www-data EUID=0 PID=161300 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: postgres
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres
[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=ed3a3df2-33b1-4c79-b551-1f69750cf7b1)
[INFO] 📦 02 setup pg instance...
[0;34m[DEADLOCK-PREVENTION][0m Deadlock prevention library loaded
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Using existing db-worker-01-postgresql environment: db-web-universe-main-dev-postgresql-worker-01.fastorder.com (10.100.1.243)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.243
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m Data dir: /data/postgresql/17/web-universe-main-dev/worker-01
[0;34m[INFO][0m Port: 5432
[0;34m[INFO][0m Hostname: db-web-universe-main-dev-postgresql-worker-01
[2026-02-05 07:19:05 UTC] USER=www-data EUID=0 PID=161408 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:05 UTC] USER=www-data EUID=0 PID=161434 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:05 UTC] USER=www-data EUID=0 PID=161466 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:05 UTC] USER=www-data EUID=0 PID=161490 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[1;33m[WARN][0m Server certificate not found at /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[0;34m[INFO][0m Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe (FastOrder Universe)
Branch: main
Env: dev
Node: worker-01
Primary CN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Alt CN: web-universe-main-dev.fastorder.com
Internal IP: 10.100.1.243 (from topology: db-worker-01-postgresql)
Shared IP: 142.93.238.16 (legacy/public, also included in SANs)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 07:19:06 UTC] USER=www-data EUID=0 PID=161551 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01 and /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:06 UTC] USER=www-data EUID=0 PID=161560 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-02-05 07:19:06 UTC] USER=www-data EUID=0 PID=161570 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-161498
[2026-02-05 07:19:06 UTC] USER=www-data EUID=0 PID=161580 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-161498/ra_root.crt
[2026-02-05 07:19:06 UTC] USER=www-data EUID=0 PID=161589 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-161498/ra_root.key
[2026-02-05 07:19:06 UTC] USER=www-data EUID=0 PID=161610 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-161498/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[2026-02-05 07:19:07 UTC] USER=www-data EUID=0 PID=161663 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-161498/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 07:19:08 UTC] USER=www-data EUID=0 PID=161674 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-161498/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 07:19:08 UTC] USER=www-data EUID=0 PID=161683 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-02-05 07:19:08 UTC] USER=www-data EUID=0 PID=161692 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-161498/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:08 UTC] USER=www-data EUID=0 PID=161701 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:08 UTC] USER=www-data EUID=0 PID=161710 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:08 UTC] USER=www-data EUID=0 PID=161719 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 07:19:08 UTC] USER=www-data EUID=0 PID=161730 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 07:19:08 UTC] USER=www-data EUID=0 PID=161739 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 07:19:08 UTC] USER=www-data EUID=0 PID=161748 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 07:19:08 UTC] USER=www-data EUID=0 PID=161757 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 07:19:08 UTC] USER=www-data EUID=0 PID=161775 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:08 UTC] USER=www-data EUID=0 PID=161784 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔍 Verifying certificate...
Certificate details:
Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
--
X509v3 Subject Alternative Name:
DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01, DNS:localhost, IP Address:10.100.1.243, IP Address:127.0.0.1, IP Address:142.93.238.16
X509v3 Subject Key Identifier:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Node: worker-01
Primary CN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-worker-01.service
3. Test SSL connection:
psql "host=db-web-universe-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-02-05 07:19:08 UTC] USER=www-data EUID=0 PID=161813 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 07:19:08 UTC] USER=www-data EUID=0 PID=161846 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[0;32m[OK][0m mTLS certificates OK (server cert + client certs verified) and keys secured
[0;34m[INFO][0m Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-02-05 07:19:08 UTC] USER=www-data EUID=0 PID=161868 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 07:19:08 UTC] USER=www-data EUID=0 PID=161902 ACTION=passthru ARGS=systemctl stop postgresql
[1;33m[WARN][0m Cleaning stale socket directory /var/run/postgresql-web-universe-main-dev-worker-01
[2026-02-05 07:19:09 UTC] USER=www-data EUID=0 PID=161936 ACTION=fsop ARGS=rm -rf /var/run/postgresql-web-universe-main-dev-worker-01
[0;32m[OK][0m No conflicting Postgres left on port 5432
[0;32m[OK][0m Using postgres password from vault provider
[2026-02-05 07:19:11 UTC] USER=www-data EUID=0 PID=162065 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.Sb8D4Z
[2026-02-05 07:19:11 UTC] USER=www-data EUID=0 PID=162095 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.Sb8D4Z
[2026-02-05 07:19:11 UTC] USER=www-data EUID=0 PID=162133 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev
[2026-02-05 07:19:11 UTC] USER=www-data EUID=0 PID=162163 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev
[2026-02-05 07:19:11 UTC] USER=www-data EUID=0 PID=162193 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/web-universe-main-dev
[0;34m[INFO][0m Initializing cluster in /data/postgresql/17/web-universe-main-dev/worker-01 (SCRAM; pwfile)
[1;33m[WARN][0m Removing existing data directory: /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 07:19:11 UTC] USER=www-data EUID=0 PID=162216 ACTION=fsop ARGS=rm -rf /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 07:19:12 UTC] USER=www-data EUID=0 PID=162267 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 07:19:12 UTC] USER=www-data EUID=0 PID=162301 ACTION=fsop ARGS=chmod 700 /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 07:19:12 UTC] USER=www-data EUID=0 PID=162322 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-web-universe-main-dev-worker-01
[2026-02-05 07:19:12 UTC] USER=www-data EUID=0 PID=162344 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-web-universe-main-dev-worker-01
[2026-02-05 07:19:12 UTC] USER=www-data EUID=0 PID=162368 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-web-universe-main-dev-worker-01
[2026-02-05 07:19:12 UTC] USER=www-data EUID=0 PID=162377 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/web-universe-main-dev/worker-01 --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.Sb8D4Z
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /data/postgresql/17/web-universe-main-dev/worker-01 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
/usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/web-universe-main-dev/worker-01 -l logfile start
[0;32m[OK][0m initdb complete
[0;34m[INFO][0m Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[0;32m[OK][0m postgresql.conf updated successfully
[0;34m[INFO][0m Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-02-05 07:19:14 UTC] USER=www-data EUID=0 PID=162493 ACTION=fsop ARGS=cp /tmp/tmp.JJZ3vY9TU3 /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
[0;32m[OK][0m pg_hba.conf updated
[0;34m[INFO][0m Creating systemd unit: /etc/systemd/system/postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 07:19:15 UTC] USER=www-data EUID=0 PID=162574 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.laqJwB /etc/systemd/system/postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 07:19:15 UTC] USER=www-data EUID=0 PID=162595 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@web-universe-main-dev-worker-01.service
[0;32m[OK][0m systemd unit written
[2026-02-05 07:19:15 UTC] USER=www-data EUID=0 PID=162621 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 07:19:15 UTC] USER=www-data EUID=0 PID=162653 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 07:19:16 UTC] USER=www-data EUID=0 PID=162679 ACTION=passthru ARGS=systemctl daemon-reload
[0;34m[INFO][0m Starting PostgreSQL instance...
[2026-02-05 07:19:17 UTC] USER=www-data EUID=0 PID=162823 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-worker-01.service
[0;34m[INFO][0m Waiting for ACTIVE (systemd)…
[2026-02-05 07:19:17 UTC] USER=www-data EUID=0 PID=162887 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-worker-01.service
[0;32m[OK][0m Service ACTIVE
[0;34m[INFO][0m Waiting for port 5432 bind…
[0;32m[OK][0m Port bound
[0;34m[INFO][0m Waiting pg_isready (socket)…
[0;32m[OK][0m Readiness via socket OK
[0;34m[INFO][0m Waiting pg_isready (TCP db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432)…
[0;32m[OK][0m Startup sequence complete
[0;34m[INFO][0m Validating core security GUCs (via local socket)…
[0;32m[OK][0m Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[0;34m[INFO][0m Provisioning application database and Debezium role (if not exists)...
[0;34m[INFO][0m Checking if database fastorder_web_universe_main_dev_db exists...
[0;34m[INFO][0m DB check result: exit_code=0, output='[2026-02-05 07:19:19 UTC] USER=www-data EUID=0 PID=163110 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_web_universe_main_dev_db''
[0;34m[INFO][0m Creating database fastorder_web_universe_main_dev_db...
[2026-02-05 07:19:19 UTC] USER=www-data EUID=0 PID=163135 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_web_universe_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[0;32m[OK][0m Database fastorder_web_universe_main_dev_db created
[0;34m[INFO][0m Checking if role debezium_user exists...
[0;34m[INFO][0m Role check result: exit_code=0, output='[2026-02-05 07:19:19 UTC] USER=www-data EUID=0 PID=163159 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[0;34m[INFO][0m Creating role debezium_user...
[2026-02-05 07:19:19 UTC] USER=www-data EUID=0 PID=163191 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'dLJ1OWcRn4Z94RPxFdb8dy9w';
CREATE ROLE
[0;32m[OK][0m Role debezium_user created
[2026-02-05 07:19:19 UTC] USER=www-data EUID=0 PID=163218 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_web_universe_main_dev_db" TO debezium_user;
GRANT
[0;32m[OK][0m Application DB (fastorder_web_universe_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[0;34m[INFO][0m Applying connection and memory optimizations...
[0;34m[INFO][0m Current settings: max_connections=100, work_mem=4MB
[0;34m[INFO][0m Target settings (worker): max_connections=100, work_mem=8MB
[2026-02-05 07:19:20 UTC] USER=www-data EUID=0 PID=163299 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 100;
ALTER SYSTEM
[2026-02-05 07:19:20 UTC] USER=www-data EUID=0 PID=163322 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-02-05 07:19:20 UTC] USER=www-data EUID=0 PID=163351 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
pg_reload_conf
----------------
t
(1 row)
[0;32m[OK][0m Settings applied to postgresql.auto.conf
[2026-02-05 07:19:20 UTC] USER=www-data EUID=0 PID=163378 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/worker-01/standby.signal
[0;34m[INFO][0m Service recently started (3s ago) - restarting to apply max_connections...
[0;34m[INFO][0m Stopping service...
[2026-02-05 07:19:21 UTC] USER=www-data EUID=0 PID=163401 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-worker-01.service
[0;34m[INFO][0m Waiting for port 5432 to be released...
[0;32m[OK][0m Port 5432 released
[0;34m[INFO][0m Starting service...
[2026-02-05 07:19:24 UTC] USER=www-data EUID=0 PID=163529 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 07:19:30 UTC] USER=www-data EUID=0 PID=163780 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-worker-01.service
[0;32m[OK][0m ✅ Optimization complete: max_connections=100, work_mem=8MB
[1;33m[WARN][0m ⚠️ Skipping synchronous replication configuration - no standbys connected yet
[0;34m[INFO][0m Synchronous replication will be enabled automatically when standbys connect
[0;34m[INFO][0m This prevents write operations from hanging during initial setup
[0;34m[INFO][0m Expected standbys: 3
[0;34m[INFO][0m Setting postgres password via centralized script... for worker-01
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: worker-01[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-worker-01[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mInitial setup: Using password from initdb[0m
[0;32m✓ PostgreSQL password already set during initdb[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/worker-01
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/worker-01[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials worker-01[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
[0;32m[OK][0m Password set and persisted
[0;34m[INFO][0m Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.243
[0;34m[INFO][0m Primary hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[0;34m[INFO][0m Adding /etc/hosts entry for worker-01...
[0;34m[INFO][0m db-web-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.243
[0;34m[INFO][0m ✅ db-web-universe-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
10.100.1.243 db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[0;32m[OK][0m PostgreSQL 'web-universe-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt \
sslcert=/home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt \
sslkey=/home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key \
host=db-web-universe-main-dev-postgresql-worker-01 port=5432 dbname=postgres user=postgres"
File been compeleted perfectly: 02-setup-pg-instance
[0;34m[INFO][0m Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: PostgreSQL
[INFO] Identifier: web-universe-main-dev-postgresql-worker-01
[INFO] Identifier Parent: worker-01
[INFO] IP: 10.100.1.243
[INFO] Port: 5432
[INFO] FQDN: db-web-universe-main-dev-postgresql-worker-01
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 84e4185a-2ef1-49c1-8d2a-841d077f036b
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[0;32m[OK][0m PostgreSQL node registered to observability API
[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=ed3a3df2-33b1-4c79-b551-1f69750cf7b1)
[INFO] 📦 03 role...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 07:19:39 UTC] USER=www-data EUID=0 PID=164486 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/worker-01/standby.signal
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: debezium_user
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: worker-01
User (CN): debezium_user
Hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164773 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-debezium_user
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164783 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164792 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-debezium_user/ra_root.key
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164801 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164810 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164835 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164862 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164873 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164906 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164925 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164936 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164945 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164954 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164963 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164972 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164981 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164990 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=164999 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 07:19:41 UTC] USER=www-data EUID=0 PID=165010 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 07:19:42 UTC] USER=www-data EUID=0 PID=165019 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:42 UTC] USER=www-data EUID=0 PID=165050 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:42 UTC] USER=www-data EUID=0 PID=165079 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:42 UTC] USER=www-data EUID=0 PID=165088 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:42 UTC] USER=www-data EUID=0 PID=165097 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:42 UTC] USER=www-data EUID=0 PID=165107 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:42 UTC] USER=www-data EUID=0 PID=165116 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:42 UTC] USER=www-data EUID=0 PID=165125 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 07:19:42 UTC] USER=www-data EUID=0 PID=165134 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 07:19:42 UTC] USER=www-data EUID=0 PID=165143 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:42 UTC] USER=www-data EUID=0 PID=165152 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:19:42 UTC] USER=www-data EUID=0 PID=165164 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 07:19:42 UTC] USER=www-data EUID=0 PID=165176 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 07:19:42 UTC] USER=www-data EUID=0 PID=165191 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 07:19:42 UTC] USER=www-data EUID=0 PID=165204 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:42 UTC] USER=www-data EUID=0 PID=165215 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:42 UTC] USER=www-data EUID=0 PID=165224 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165234 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165243 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165252 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165262 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165273 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165283 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165292 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165303 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165312 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165321 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165331 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165344 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165353 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165363 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165372 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165382 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165403 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 07:19:43 UTC] USER=www-data EUID=0 PID=165414 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165434 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165450 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165459 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165472 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165487 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165496 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165508 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165517 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165527 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165548 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165561 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165570 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165579 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165588 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165597 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165608 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165618 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165627 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165637 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165646 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165655 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 07:19:44 UTC] USER=www-data EUID=0 PID=165664 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:19:45 UTC] USER=www-data EUID=0 PID=165682 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:19:45 UTC] USER=www-data EUID=0 PID=165691 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: debezium_user
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U debezium_user -d postgres
🔐 Generating replicator client certificate for worker-01...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: replicator
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: worker-01
User (CN): replicator
Hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:19:45 UTC] USER=www-data EUID=0 PID=165756 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-02-05 07:19:45 UTC] USER=www-data EUID=0 PID=165765 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-02-05 07:19:45 UTC] USER=www-data EUID=0 PID=165775 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-02-05 07:19:45 UTC] USER=www-data EUID=0 PID=165784 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=165809 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=165833 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=165845 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=165865 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=165895 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=165909 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=165919 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=165928 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=165938 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=165947 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=165956 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=165965 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=165974 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=165983 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=165992 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=166001 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=166020 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=166046 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=166055 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=166064 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:46 UTC] USER=www-data EUID=0 PID=166075 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166084 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166093 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166102 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166114 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166137 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166155 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166164 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166174 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166184 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166193 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166202 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166211 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166220 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166229 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166247 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166256 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166265 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166274 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166284 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166296 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166306 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166315 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166324 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166333 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166342 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166351 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166371 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166380 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:19:47 UTC] USER=www-data EUID=0 PID=166389 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166398 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166417 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166427 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166436 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166445 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166454 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166463 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166472 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166481 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166491 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166501 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166510 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166519 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166528 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166538 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166557 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
✅ Symlinked ca.pem
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166575 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
✅ Symlinked client-key.pem
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166593 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166611 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:19:48 UTC] USER=www-data EUID=0 PID=166625 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: replicator
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres
✅ Replicator certificate generated for worker-01
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: worker-01[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-worker-01[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mGenerating new secure password...[0m
[0;34mUser fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: fastorder_admin_gd[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials worker-01[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: worker-01/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: fastorder_admin_gd
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: worker-01
User (CN): fastorder_admin_gd
Hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:19:57 UTC] USER=www-data EUID=0 PID=167193 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-fastorder_admin_gd
[2026-02-05 07:19:57 UTC] USER=www-data EUID=0 PID=167202 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-02-05 07:19:57 UTC] USER=www-data EUID=0 PID=167211 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
[2026-02-05 07:19:57 UTC] USER=www-data EUID=0 PID=167220 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-02-05 07:19:57 UTC] USER=www-data EUID=0 PID=167229 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:58 UTC] USER=www-data EUID=0 PID=167248 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:58 UTC] USER=www-data EUID=0 PID=167257 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:58 UTC] USER=www-data EUID=0 PID=167266 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 07:19:58 UTC] USER=www-data EUID=0 PID=167275 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 07:19:58 UTC] USER=www-data EUID=0 PID=167284 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:58 UTC] USER=www-data EUID=0 PID=167293 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:19:58 UTC] USER=www-data EUID=0 PID=167303 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:19:58 UTC] USER=www-data EUID=0 PID=167312 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 07:19:58 UTC] USER=www-data EUID=0 PID=167321 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 07:19:58 UTC] USER=www-data EUID=0 PID=167355 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:19:58 UTC] USER=www-data EUID=0 PID=167364 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 07:19:59 UTC] USER=www-data EUID=0 PID=167382 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:59 UTC] USER=www-data EUID=0 PID=167392 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:19:59 UTC] USER=www-data EUID=0 PID=167406 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 07:19:59 UTC] USER=www-data EUID=0 PID=167417 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:19:59 UTC] USER=www-data EUID=0 PID=167426 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 07:19:59 UTC] USER=www-data EUID=0 PID=167436 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 07:19:59 UTC] USER=www-data EUID=0 PID=167445 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:59 UTC] USER=www-data EUID=0 PID=167455 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:59 UTC] USER=www-data EUID=0 PID=167487 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:59 UTC] USER=www-data EUID=0 PID=167496 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:59 UTC] USER=www-data EUID=0 PID=167506 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:19:59 UTC] USER=www-data EUID=0 PID=167515 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:59 UTC] USER=www-data EUID=0 PID=167529 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:19:59 UTC] USER=www-data EUID=0 PID=167548 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 07:19:59 UTC] USER=www-data EUID=0 PID=167561 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:19:59 UTC] USER=www-data EUID=0 PID=167599 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 07:19:59 UTC] USER=www-data EUID=0 PID=167609 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:20:00 UTC] USER=www-data EUID=0 PID=167619 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:20:00 UTC] USER=www-data EUID=0 PID=167630 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:20:00 UTC] USER=www-data EUID=0 PID=167639 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:20:00 UTC] USER=www-data EUID=0 PID=167653 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:20:00 UTC] USER=www-data EUID=0 PID=167665 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:20:00 UTC] USER=www-data EUID=0 PID=167674 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 07:20:00 UTC] USER=www-data EUID=0 PID=167683 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 07:20:00 UTC] USER=www-data EUID=0 PID=167693 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:20:00 UTC] USER=www-data EUID=0 PID=167704 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:20:00 UTC] USER=www-data EUID=0 PID=167713 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:20:00 UTC] USER=www-data EUID=0 PID=167722 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 07:20:00 UTC] USER=www-data EUID=0 PID=167731 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 07:20:00 UTC] USER=www-data EUID=0 PID=167743 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:20:00 UTC] USER=www-data EUID=0 PID=167762 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:20:00 UTC] USER=www-data EUID=0 PID=167771 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:20:00 UTC] USER=www-data EUID=0 PID=167785 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:20:00 UTC] USER=www-data EUID=0 PID=167794 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:20:01 UTC] USER=www-data EUID=0 PID=167820 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:20:01 UTC] USER=www-data EUID=0 PID=167830 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:20:01 UTC] USER=www-data EUID=0 PID=167839 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:20:01 UTC] USER=www-data EUID=0 PID=167851 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 07:20:01 UTC] USER=www-data EUID=0 PID=167860 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 07:20:01 UTC] USER=www-data EUID=0 PID=167870 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:20:01 UTC] USER=www-data EUID=0 PID=167882 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:20:01 UTC] USER=www-data EUID=0 PID=167892 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:20:01 UTC] USER=www-data EUID=0 PID=167940 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:20:01 UTC] USER=www-data EUID=0 PID=167966 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:20:02 UTC] USER=www-data EUID=0 PID=167979 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 07:20:02 UTC] USER=www-data EUID=0 PID=167988 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 07:20:02 UTC] USER=www-data EUID=0 PID=167997 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:20:02 UTC] USER=www-data EUID=0 PID=168008 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:20:02 UTC] USER=www-data EUID=0 PID=168017 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:20:02 UTC] USER=www-data EUID=0 PID=168031 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 07:20:02 UTC] USER=www-data EUID=0 PID=168052 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 07:20:02 UTC] USER=www-data EUID=0 PID=168062 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:20:02 UTC] USER=www-data EUID=0 PID=168078 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:20:02 UTC] USER=www-data EUID=0 PID=168087 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
✅ Symlinked ca.pem
[2026-02-05 07:20:02 UTC] USER=www-data EUID=0 PID=168108 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 07:20:02 UTC] USER=www-data EUID=0 PID=168120 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 07:20:03 UTC] USER=www-data EUID=0 PID=168131 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:20:03 UTC] USER=www-data EUID=0 PID=168145 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:20:03 UTC] USER=www-data EUID=0 PID=168171 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: fastorder_admin_gd
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U fastorder_admin_gd -d postgres
🧱 Connecting via Unix socket to create role and database...
Socket: /var/run/postgresql-web-universe-main-dev-worker-01:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️ Database fastorder_web_universe_main_dev_db already exists, skipping creation
[2026-02-05 07:20:04 UTC] USER=www-data EUID=0 PID=168267 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️ Using pg_hba.conf: /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-02-05 07:20:04 UTC] USER=www-data EUID=0 PID=168308 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-worker-01.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[0;34m=== Pre-flight Checks ===[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;32m✓[0m AWS Secrets Manager accessible
[0;34m=== Retrieving Credentials from AWS ===[0m
ℹ️ Retrieving PostgreSQL credentials for: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️ Fetching secret: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials loaded for worker-01/fastorder_admin_gd: fastorder_admin_gd@db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_web_universe_main_dev_db
[0;32m✓[0m Credentials retrieved: fastorder_admin_gd@db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_web_universe_main_dev_db
[0;34m╔════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Test Suite (AWS Secrets MGR) ║[0m
[0;34m╚════════════════════════════════════════════╝[0m
[0;34m=== PostgreSQL Authentication Test ===[0m
[0;31m✗[0m PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-web-universe-main-dev-postgresql-worker-01.fastorder.com" (10.100.1.243), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 07:20:09 UTC] USER=www-data EUID=0 PID=168585 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/worker-01/standby.signal
── fast setup ─────────────────────────────────────────────
NAME : web-universe-main-dev
IDENTIFIER : worker-01
PG HOST : db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432
ROLE : debezium_user
DB : fastorder_web_universe_main_dev_db
SCHEMA : web
AUTH MODE : scram (scram=password over TLS | cert=mTLS)
SUBNET ALLOW: 10.201.0.0/16
CONNECT /32 : 142.93.238.16
SSL DIR : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
DNS → 10.100.1.243
CA : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: worker-01[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-worker-01[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mGenerating new secure password...[0m
[0;34mUser debezium_user does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: debezium_user[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials worker-01[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: worker-01/debezium_user
✓ Retrieved password from secrets vault
password : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️ Role debezium_user exists, updating
[2026-02-05 07:20:17 UTC] USER=www-data EUID=0 PID=169114 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️ Database fastorder_web_universe_main_dev_db already exists
[2026-02-05 07:20:17 UTC] USER=www-data EUID=0 PID=169142 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d fastorder_web_universe_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️ Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
🔐 Creating replicator role for worker-01...
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
NAME : web-universe-main-dev
IDENTIFIER : worker-01
PG HOST : db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432
ROLE : replicator
SSL DIR : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
DNS → 10.100.1.243
CA : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️ Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE: Creating role: replicator with password
SET
CREATE ROLE
✅ Replicator role ensured with password authentication.
ℹ️ Password stored in: AWS Secrets Manager
Secret name: fastorder/db/web/universe/main/dev/postgresql/replicator
🔄 MIGRATION PATH: Password → Certificate Authentication
Current: SCRAM-SHA-256 password auth (production-ready)
Future: Certificate-based auth (requires CA automation)
To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
and configure standby to use SSL certificates instead of password
🎉 Done.
✅ Replicator role created for worker-01
[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=ed3a3df2-33b1-4c79-b551-1f69750cf7b1)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
ℹ️ Service-specific setup (web) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)
🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=web
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
🔍 DEBUG_CHECKPOINT_03: Found service-specific run.sh for: web
[DEBUG] Tracking substep start: steps/01-install/steps/web (RUN_UUID=ed3a3df2-33b1-4c79-b551-1f69750cf7b1)
[INFO] 🔸 Service: web (service-specific)
🔍 DEBUG_CHECKPOINT_04: Executing service-specific: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Web Service Schema Provisioning
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service: web
[INFO] Zone: universe
[INFO] Branch: main
[INFO] Environment: dev
[INFO] Identifier: coordinator
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] 📦 Running: 01-init-schema.sh...
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Initializing config schema
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Identifier: coordinator
Database: fastorder_web_universe_main_dev_db
Host: db-web-universe-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking if config schema exists...
[INFO] Config schema already exists - checking tables...
[OK] Config schema with 3 tables already exists - skipping
[OK] ✓ 01-init-schema.sh completed
[INFO] ═══════════════════════════════════════════════════════════════
[OK] Web service schema provisioning complete
[INFO] ═══════════════════════════════════════════════════════════════
[0;32m✓[0m ✅ Worker worker-01 setup completed
Skipping standbys (PG_WORKERS_STANDBY_NUM=0)
[0;32m✓[0m ✅ PostgreSQL installation completed
[0;34m[INFO][0m Discovering additional setup steps...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 02-pg-bouncer.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up PgBouncer connection pooling...
[2026-02-05 07:20:26 UTC] USER=www-data EUID=0 PID=169553 ACTION=fsop ARGS=rm -f /tmp/pgbouncer-ip.service /tmp/pgbouncer.service
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;32m✓ [SECRETS][0m Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[0;34m[SECRETS][0m Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[0;34m[SECRETS][0m Search (build_es_secret_name, get/set_es_credentials_to_vault)
[0;34m[SECRETS][0m Backups (build_backup_path)
[0;34m[SECRETS][0m Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[0;34m[INFO][0m Checking for existing PgBouncer application environment in topology …
[0;32m[OK][0m Using existing PgBouncer environment:
[0;34m[INFO][0m IP: 10.100.1.244
[0;34m[INFO][0m FQDN: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[0;34m[INFO][0m Domain: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[0;34m[INFO][0m Ensuring /etc/hosts entry for db-web-universe-main-dev-postgresql-bouncer.fastorder.com …
[0;32m[OK][0m /etc/hosts already contains entry for db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[1;33m[WARN][0m IP 10.100.1.244 is assigned to multiple interfaces:
inet 10.100.1.242/32 scope global lo
valid_lft forever preferred_lft forever
inet 10.100.1.244/32 scope global lo:pgbouncer
--
inet 10.100.1.243/32 scope global eth0
valid_lft forever preferred_lft forever
inet 10.100.1.244/32 scope global eth0:244
[1;33m[WARN][0m This may cause routing issues
[0;34m[INFO][0m Final verification of /etc/hosts entry for db-web-universe-main-dev-postgresql-bouncer.fastorder.com …
[0;32m[OK][0m /etc/hosts correctly maps db-web-universe-main-dev-postgresql-bouncer.fastorder.com to 10.100.1.244
[0;32m[OK][0m PgBouncer IP 10.100.1.244 already correctly bound to lo:pgbouncer
[2026-02-05 07:20:26 UTC] USER=www-data EUID=0 PID=169697 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 07:20:27 UTC] USER=www-data EUID=0 PID=169859 ACTION=passthru ARGS=systemctl restart pgbouncer-ip@web-universe-main-dev.service
Job for pgbouncer-ip@web-universe-main-dev.service failed because the control process exited with error code.
See "systemctl status pgbouncer-ip@web-universe-main-dev.service" and "journalctl -xeu pgbouncer-ip@web-universe-main-dev.service" for details.
[2026-02-05 07:20:28 UTC] USER=www-data EUID=0 PID=169869 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer-ip@web-universe-main-dev.service
[1;33m[WARN][0m pgbouncer-ip@web-universe-main-dev.service is not active
[1;33m[WARN][0m Check status: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@web-universe-main-dev.service
[2026-02-05 07:20:28 UTC] USER=www-data EUID=0 PID=169913 ACTION=fsop ARGS=mkdir -p /run/pgbouncer/web-universe-main-dev
[2026-02-05 07:20:28 UTC] USER=www-data EUID=0 PID=169922 ACTION=fsop ARGS=mkdir -p /var/log/pgbouncer/web-universe-main-dev
[2026-02-05 07:20:28 UTC] USER=www-data EUID=0 PID=169931 ACTION=fsop ARGS=chmod 750 /etc/pgbouncer/web-universe-main-dev
[2026-02-05 07:20:28 UTC] USER=www-data EUID=0 PID=169941 ACTION=fsop ARGS=chmod 750 /run/pgbouncer/web-universe-main-dev
[2026-02-05 07:20:28 UTC] USER=www-data EUID=0 PID=169952 ACTION=fsop ARGS=chmod 750 /var/log/pgbouncer/web-universe-main-dev
[2026-02-05 07:20:28 UTC] USER=www-data EUID=0 PID=169961 ACTION=fsop ARGS=chown root:postgres /etc/pgbouncer/web-universe-main-dev
[2026-02-05 07:20:28 UTC] USER=www-data EUID=0 PID=169975 ACTION=fsop ARGS=chown postgres:postgres /run/pgbouncer/web-universe-main-dev
[2026-02-05 07:20:28 UTC] USER=www-data EUID=0 PID=169984 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbouncer/web-universe-main-dev
[0;34m[INFO][0m Generating pgbouncer_admin client certificates...
[0;34m[INFO][0m ⏳ This may take 30-60 seconds...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: pgbouncer_admin
Identifier: pgbouncer
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: pgbouncer
User (CN): pgbouncer_admin
Hostname: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:20:28 UTC] USER=www-data EUID=0 PID=170075 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-pgbouncer-pgbouncer_admin
[2026-02-05 07:20:28 UTC] USER=www-data EUID=0 PID=170085 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170096 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170105 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170114 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = pgbouncer_admin
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170130 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170145 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170154 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170163 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170172 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170181 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170190 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170204 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170214 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170223 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170232 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170244 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170253 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170262 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170276 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170289 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 07:20:29 UTC] USER=www-data EUID=0 PID=170298 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 07:20:30 UTC] USER=www-data EUID=0 PID=170316 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 07:20:30 UTC] USER=www-data EUID=0 PID=170325 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 07:20:30 UTC] USER=www-data EUID=0 PID=170336 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:30 UTC] USER=www-data EUID=0 PID=170371 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:20:30 UTC] USER=www-data EUID=0 PID=170389 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:30 UTC] USER=www-data EUID=0 PID=170398 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:30 UTC] USER=www-data EUID=0 PID=170407 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 07:20:30 UTC] USER=www-data EUID=0 PID=170417 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 07:20:30 UTC] USER=www-data EUID=0 PID=170428 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 07:20:30 UTC] USER=www-data EUID=0 PID=170438 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 07:20:30 UTC] USER=www-data EUID=0 PID=170447 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 07:20:30 UTC] USER=www-data EUID=0 PID=170456 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 07:20:30 UTC] USER=www-data EUID=0 PID=170467 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 07:20:30 UTC] USER=www-data EUID=0 PID=170477 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:30 UTC] USER=www-data EUID=0 PID=170487 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:30 UTC] USER=www-data EUID=0 PID=170496 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:20:30 UTC] USER=www-data EUID=0 PID=170507 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170518 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170532 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170542 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170553 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170565 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170574 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170589 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170599 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170609 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170623 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170642 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170651 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170665 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170678 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170687 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170696 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170705 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170714 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170723 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170732 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 07:20:31 UTC] USER=www-data EUID=0 PID=170741 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170751 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170764 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170774 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170783 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170800 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170809 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170818 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170827 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170836 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170845 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170854 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170863 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170872 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170894 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170903 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170912 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
✅ Symlinked client-cert.pem
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170931 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170944 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170953 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170963 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:20:32 UTC] USER=www-data EUID=0 PID=170980 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: pgbouncer_admin
Node: pgbouncer
FQDN: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-bouncer.fastorder.com -U pgbouncer_admin -d postgres
[0;32m[OK][0m mTLS client certificate present: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[0;34m[INFO][0m Creating symlinks to canonical certificates in /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend...
[2026-02-05 07:20:33 UTC] USER=www-data EUID=0 PID=171015 ACTION=fsop ARGS=mkdir -p /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend
[2026-02-05 07:20:33 UTC] USER=www-data EUID=0 PID=171033 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-02-05 07:20:33 UTC] USER=www-data EUID=0 PID=171042 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[2026-02-05 07:20:33 UTC] USER=www-data EUID=0 PID=171051 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt
[0;34m[INFO][0m Creating coordinator CA symlink for PostgreSQL server verification...
[0;34m[INFO][0m Verifying canonical certificate permissions...
[2026-02-05 07:20:33 UTC] USER=www-data EUID=0 PID=171069 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 07:20:33 UTC] USER=www-data EUID=0 PID=171078 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 07:20:33 UTC] USER=www-data EUID=0 PID=171087 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 07:20:33 UTC] USER=www-data EUID=0 PID=171097 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[0;32m[OK][0m Backend certificate symlinks created in /etc/ssl
[0;32m[OK][0m Coordinator CA symlink created for server verification
[0;32m[OK][0m Certificates already in canonical location - no symlinks needed
[2026-02-05 07:20:33 UTC] USER=www-data EUID=0 PID=171108 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
[2026-02-05 07:20:33 UTC] USER=www-data EUID=0 PID=171127 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
[2026-02-05 07:20:33 UTC] USER=www-data EUID=0 PID=171136 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 07:20:33 UTC] USER=www-data EUID=0 PID=171145 ACTION=fsop ARGS=test -r /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m PgBouncer will use PostgreSQL coordinator CA: /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;32m[OK][0m PostgreSQL coordinator at db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432 is reachable
[0;34m[INFO][0m Dumping SCRAM secrets from coordinator for PgBouncer auth_file …
[2026-02-05 07:20:33 UTC] USER=www-data EUID=0 PID=171175 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/userlist.txt
[2026-02-05 07:20:33 UTC] USER=www-data EUID=0 PID=171187 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;32m[OK][0m Auth file written: /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;34m[INFO][0m Retrieved password from vault for pgbouncer_admin
[0;34m[INFO][0m Ensuring PgBouncer admin role 'pgbouncer_admin' exists in Postgres (coordinator) …
[0;32m[OK][0m Role pgbouncer_admin created/updated successfully
[0;34m[SECRETS][0m Setting credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/pgbouncer_admin
[0;32m✓ [SECRETS][0m Credentials updated in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/pgbouncer_admin
[0;34m[INFO][0m ✅ PgBouncer admin password stored in centralized secrets vault
[0;34m[INFO][0m Re-fetching SCRAM secrets after role creation to ensure pgbouncer_admin is included …
[2026-02-05 07:20:39 UTC] USER=www-data EUID=0 PID=171353 ACTION=fsop ARGS=cp /tmp/tmp.pjsu9iKEGA /etc/pgbouncer/web-universe-main-dev/userlist.txt
[2026-02-05 07:20:39 UTC] USER=www-data EUID=0 PID=171363 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;32m[OK][0m Auth file updated with pgbouncer_admin SCRAM hash
[0;34m[INFO][0m Auth file contains [2026-02-05 07:20:39 UTC] USER=www-data EUID=0 PID=171395 ACTION=passthru ARGS=bash -c wc -l < '/etc/pgbouncer/web-universe-main-dev/userlist.txt'
4 user(s)
[0;32m[OK][0m Admin 'pgbouncer_admin' password generated and saved
[0;34m[INFO][0m Configuring PostgreSQL to prevent Citus metadata sync hangs...
ALTER ROLE
[0;32m[OK][0m Disabled Citus metadata sync for pgbouncer_admin
[0;34m[INFO][0m Verifying application database fastorder_web_universe_main_dev_db exists...
[0;32m[OK][0m ✓ Database fastorder_web_universe_main_dev_db exists
[0;34m[INFO][0m Granting permissions to pgbouncer_admin on fastorder_web_universe_main_dev_db...
GRANT
[0;32m[OK][0m ✓ Granted CONNECT on fastorder_web_universe_main_dev_db to pgbouncer_admin
GRANT
[0;32m[OK][0m ✓ Granted USAGE on schema public to pgbouncer_admin
GRANT
[0;32m[OK][0m ✓ Granted SELECT on all tables to pgbouncer_admin
ALTER DATABASE
[0;32m[OK][0m Set synchronous_commit=local for fastorder_web_universe_main_dev_db
[0;34m[INFO][0m Ensuring pg_hba.conf entry for pgbouncer_admin …
[0;34m[INFO][0m Adding pg_hba.conf entries for pgbouncer_admin with cert auth …
[0;32m[OK][0m pg_hba.conf updated and PostgreSQL configuration reloaded
[2026-02-05 07:20:40 UTC] USER=unknown EUID=33 PID=171555 ACTION=-u ARGS=postgres bash
ERROR: Invalid or unauthorized action: -u
[1;33m[WARN][0m pg_hba.conf entry may not have loaded correctly
[0;34m[INFO][0m Writing /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini …
[2026-02-05 07:20:41 UTC] USER=www-data EUID=0 PID=171602 ACTION=fsop ARGS=cp /tmp/tmp.K89uhx4aEC /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
[2026-02-05 07:20:41 UTC] USER=www-data EUID=0 PID=171614 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
[2026-02-05 07:20:41 UTC] USER=www-data EUID=0 PID=171641 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbouncer/web-universe-main-dev /run/pgbouncer/web-universe-main-dev /var/log/pgbouncer/web-universe-main-dev
[2026-02-05 07:20:41 UTC] USER=www-data EUID=0 PID=171650 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;32m[OK][0m pgbouncer.ini ready
[0;34m[INFO][0m Verifying TLS settings in pgbouncer.ini:
[2026-02-05 07:20:41 UTC] USER=www-data EUID=0 PID=171663 ACTION=fsop ARGS=grep -E (client_tls_sslmode|server_tls) /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
client_tls_sslmode = verify-full
server_tls_sslmode = verify-full
server_tls_ca_file = /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
server_tls_cert_file = /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
server_tls_key_file = /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;34m[INFO][0m Verifying PgBouncer server certificate files:
[2026-02-05 07:20:41 UTC] USER=www-data EUID=0 PID=171672 ACTION=fsop ARGS=test -r /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[0;32m[OK][0m Server cert readable by postgres: /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-02-05 07:20:41 UTC] USER=www-data EUID=0 PID=171681 ACTION=fsop ARGS=test -r /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;32m[OK][0m Server key readable by postgres: /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;34m[INFO][0m Verifying coordinator CA certificate:
[2026-02-05 07:20:41 UTC] USER=www-data EUID=0 PID=171690 ACTION=fsop ARGS=test -r /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;32m[OK][0m Coordinator CA readable by postgres: /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m Preflight: stopping any conflicting PgBouncer on 6432 …
[2026-02-05 07:20:41 UTC] USER=www-data EUID=0 PID=171699 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer.service
[2026-02-05 07:20:41 UTC] USER=www-data EUID=0 PID=171708 ACTION=passthru ARGS=systemctl stop pgbouncer@web-universe-main-dev.service
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json?all=1": dial unix /var/run/docker.sock: connect: permission denied
[2026-02-05 07:20:44 UTC] USER=www-data EUID=0 PID=171822 ACTION=passthru ARGS=systemctl daemon-reload
[0;32m[OK][0m systemd unit installed: pgbouncer@web-universe-main-dev.service
[0;34m[INFO][0m Running pre-flight IP conflict check for 10.100.1.244:6432 …
[1;33m[WARN][0m IP conflict checker not found at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/check-ip-conflicts.sh
[1;33m[WARN][0m Skipping pre-flight check - conflicts may occur
[0;34m[INFO][0m Starting PgBouncer (web-universe-main-dev) …
[2026-02-05 07:20:45 UTC] USER=www-data EUID=0 PID=171952 ACTION=passthru ARGS=systemctl restart pgbouncer@web-universe-main-dev.service
[2026-02-05 07:20:45 UTC] USER=www-data EUID=0 PID=171963 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer@web-universe-main-dev.service
[0;32m[OK][0m Service ACTIVE
[0;34m[INFO][0m Verifying auth_file before probing …
[0;34m[INFO][0m Auth file contains 4 user(s)
[1;33m[WARN][0m Auth file does NOT contain pgbouncer_admin entry - authentication will fail
[0;34m[INFO][0m Probing admin console via SSL (psql to database 'pgbouncer') …
[0;34m[INFO][0m Retrieved password from vault for admin console probe
[0;32m[OK][0m Admin console reachable (SHOW POOLS OK)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Running Comprehensive PgBouncer Verification Tests
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Password extracted: WO0D2C0d7Z... (using postgres user certificates)
[0;34m[INFO][0m Test 1/7: Admin Console - SHOW POOLS
database | user | cl_active | cl_waiting | cl_active_cancel_req | cl_waiting_cancel_req | sv_active | sv_active_cancel | sv_being_canceled | sv_idle | sv_used | sv_tested | sv_login | maxwait | maxwait_us | pool_mode | load_balance_hosts
-----------+-----------+-----------+------------+----------------------+-----------------------+-----------+------------------+-------------------+---------+---------+-----------+----------+---------+------------+-----------+--------------------
pgbouncer | pgbouncer | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | statement |
(1 row)
[0;32m[OK][0m ✓ SHOW POOLS: SUCCESS
[0;34m[INFO][0m Test 2/7: Admin Console - SHOW VERSION
[0;32m[OK][0m ✓ SHOW VERSION: PgBouncer 1.24.1
[0;34m[INFO][0m Test 3/7: Admin Console - SHOW STATS
database | total_server_assignment_count | total_xact_count | total_query_count | total_received | total_sent | total_xact_time | total_query_time | total_wait_time | total_client_parse_count | total_server_parse_count | total_bind_count | avg_server_assignment_count | avg_xact_count | avg_query_count | avg_recv | avg_sent | avg_xact_time | avg_query_time | avg_wait_time | avg_client_parse_count | avg_server_parse_count | avg_bind_count
-----------+-------------------------------+------------------+-------------------+----------------+------------+-----------------+------------------+-----------------+--------------------------+--------------------------+------------------+-----------------------------+----------------+-----------------+----------+----------+---------------+----------------+---------------+------------------------+------------------------+----------------
pgbouncer | 0 | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
(1 row)
[0;32m[OK][0m ✓ SHOW STATS: SUCCESS
[0;34m[INFO][0m Test 4/7: Admin Console - SHOW DATABASES
name | host | port | database | force_user | pool_size | min_pool_size | reserve_pool_size | server_lifetime | pool_mode | load_balance_hosts | max_connections | current_connections | max_client_connections | current_client_connections | paused | disabled
---------------------------------------------+---------------------------------------------------------------+------+------------------------------------+------------+-----------+---------------+-------------------+-----------------+-----------+--------------------+-----------------+---------------------+------------------------+----------------------------+--------+----------
fastorder_web_universe_main_dev_db | db-web-universe-main-dev-postgresql-coordinator.fastorder.com | 5432 | fastorder_web_universe_main_dev_db | | 100 | 0 | 20 | 3600 | | | 0 | 0 | 0 | 0 | 0 | 0
fastorder_web_universe_main_dev_db_worker_1 | pg-worker-01-web-universe-main-dev.fastorder.com | 5432 | fastorder_web_universe_main_dev_db | | 100 | 0 | 20 | 3600 | | | 0 | 0 | 0 | 0 | 0 | 0
fastorder_web_universe_main_dev_db_worker_2 | pg-worker-01-standby-01-web-universe-main-dev.fastorder.com | 5432 | fastorder_web_universe_main_dev_db | | 100 | 0 | 20 | 3600 | | | 0 | 0 | 0 | 0 | 0 | 0
fastorder_web_universe_main_dev_db_worker_3 | pg-worker-01-standby-02-web-universe-main-dev.fastorder.com | 5432 | fastorder_web_universe_main_dev_db | | 100 | 0 | 20 | 3600 | | | 0 | 0 | 0 | 0 | 0 | 0
fastorder_web_universe_main_dev_db_worker_4 | pg-worker-01-standby-03-web-universe-main-dev.fastorder.com | 5432 | fastorder_web_universe_main_dev_db | | 100 | 0 | 20 | 3600 | | | 0 | 0 | 0 | 0 | 0 | 0
pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | 0 | 3600 | statement | | 0 | 0 | 0 | 1 | 0 | 0
(6 rows)
[0;32m[OK][0m ✓ SHOW DATABASES: SUCCESS
[0;34m[INFO][0m Test 5/7: Admin Console - SHOW CONFIG
[0;32m[OK][0m ✓ SHOW CONFIG: SUCCESS
[0;34m[INFO][0m Key settings:
[0;34m[INFO][0m client_tls_sslmode = verify-full|disable|yes
[0;34m[INFO][0m max_client_conn = 2048|100|yes
[0;34m[INFO][0m pool_mode = transaction|session|yes
[0;34m[INFO][0m server_tls_sslmode = verify-full|prefer|yes
psql "host=db-web-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_web_universe_main_dev_db user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD connect_timeout=5 sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt sslcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt sslkey=/etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key" --no-psqlrc -Atc 'SELECT version();'
[0;34m[INFO][0m Test 6/7: Application Database - SELECT version()
[1;33m[WARN][0m ✗ Application database query: FAILED (timeout or connection issue)
[1;33m[WARN][0m If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[0;34m[INFO][0m Test 7/8: Application Database - Connection Details
[1;33m[WARN][0m ✗ Connection details: FAILED (timeout or connection issue)
[1;33m[WARN][0m If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[0;34m[INFO][0m Test 8/8: End-to-End Application Routing - Pool Verification
[0;34m[INFO][0m Running actual queries through PgBouncer to verify routing and pooling...
[1;33m[WARN][0m ✗ End-to-end routing verification: FAILED - All 3 queries failed
[1;33m[WARN][0m If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[1;33m[WARN][0m Otherwise check if database fastorder_web_universe_main_dev_db exists and user pgbouncer_admin has permissions
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verification Complete - Tests 1-5 PASSED (Admin console verified)
[1;33m[WARN][0m Tests 6-8 FAILED - Application database not accessible
[1;33m[WARN][0m This is expected if Citus is not set up yet
[1;33m[WARN][0m Run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m PgBouncer is up for web-universe-main-dev
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Connection Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Password stored in: AWS Secrets Manager (fastorder/db/web/ksa/main/dev/postgresqlweb/universe/main/dev/coordinator-pgbouncer_admin)
Current password: WO0D2C0d7ZbIdk65D10y9TaD
1. Admin Console (using IP address to avoid DNS/SSL issues):
psql "host=10.100.1.244 port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD sslmode=verify-full sslrootcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"
2. Admin Console (using hostname):
psql "host=db-web-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD sslmode=verify-full sslrootcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"
3. Application Database:
psql "host=db-web-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_web_universe_main_dev_db sslkey=/etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD sslmode=verify-full sslrootcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"
4. Using .pgpass file:
echo "db-web-universe-main-dev-postgresql-bouncer.fastorder.com:6432:*:pgbouncer_admin:WO0D2C0d7ZbIdk65D10y9TaD" >> ~/.pgpass
chmod 600 ~/.pgpass
psql -h db-web-universe-main-dev-postgresql-bouncer.fastorder.com -p 6432 -U pgbouncer_admin -d fastorder_web_universe_main_dev_db
5. Retrieve password from vault:
source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
PGPASSWORD="$(get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password')" \
psql -h 10.100.1.244 -p 6432 -U pgbouncer_admin -d pgbouncer -c "SHOW POOLS;"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Architecture
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
• Default db 'fastorder_web_universe_main_dev_db' → Citus coordinator (db-web-universe-main-dev-postgresql-coordinator.fastorder.com)
• Worker access: 'fastorder_web_universe_main_dev_db_worker_1', 'fastorder_web_universe_main_dev_db_worker_2', … (if exist)
• Client TLS: require (password auth) / verify-full (mTLS with certs)
• Server TLS: verify-full (PgBouncer validates PostgreSQL certs)
• Auth: SCRAM-SHA-256 via /etc/pgbouncer/web-universe-main-dev/userlist.txt
• Pool mode: transaction (stateless connections)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Management
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Service Status:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer@web-universe-main-dev.service
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@web-universe-main-dev.service
Logs:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@web-universe-main-dev.service -f
/usr/local/bin/fastorder-provisioning-wrapper.sh tail -f /var/log/pgbouncer/web-universe-main-dev/pgbouncer.log
Reload Config:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@web-universe-main-dev.service
Restart:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@web-universe-main-dev.service
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Files
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Config: /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
Auth file: /etc/pgbouncer/web-universe-main-dev/userlist.txt
Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
Server key: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
CA cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt
PG CA: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
Logs: /var/log/pgbouncer/web-universe-main-dev/pgbouncer.log
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Troubleshooting
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
If "SASL authentication failed":
1. Check auth file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/web-universe-main-dev/userlist.txt
2. Verify pgbouncer_admin is present with SCRAM hash
3. Get password from vault:
source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password'
4. Reload PgBouncer: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@web-universe-main-dev.service
If "no pg_hba.conf entry":
1. Check pg_hba.conf on coordinator
2. Add rule: hostssl all pgbouncer_admin 10.100.1.244/32 cert clientcert=verify-full
3. Reload PostgreSQL
To add users to PgBouncer:
1. Create user in PostgreSQL with password
2. Re-run SCRAM dump:
psql "host=db-web-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 dbname=postgres user=postgres \
sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt \
sslcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt sslkey=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key" \
-Atc "SELECT '\"' || rolname || '\" \"' || rolpassword || '\"' \
FROM pg_authid WHERE rolpassword LIKE 'SCRAM-SHA-256%' \
AND rolcanlogin ORDER BY rolname;" | command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop tee /etc/pgbouncer/web-universe-main-dev/userlist.txt
3. Reload: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@web-universe-main-dev.service
[0;34m[INFO][0m Registering PgBouncer node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: PgBouncer
[INFO] Identifier: web-universe-main-dev-pgbouncer
[INFO] Identifier Parent: postgresql
[INFO] IP: 10.100.1.244
[INFO] Port: 6432
[INFO] FQDN: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: fdc70c5f-615d-432f-8161-a7acd56ea9ed
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[0;32m[OK][0m PgBouncer node registered to observability API
[0;32m✓[0m ✅ PgBouncer setup completed
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 03-citus-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m CITUS DISTRIBUTED CLUSTER SETUP
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Phase 1: Installing Citus extension on workers...
[0;34m[INFO][0m Phase 2: Setting up coordinator and registering workers...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 📦 PHASE 1: Installing Citus extension on 1 worker(s)...
[0;34m[INFO][0m → Worker 1/1: Installing Citus on worker-01...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS CLUSTER SETUP
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m 🔧 Setting up Citus Worker...
[0;34m[INFO][0m Temporarily disabling synchronous replication for extension installation...
t
[0;34m[INFO][0m Installing Citus extension on worker...
[0;32m[OK][0m Citus extension installed on worker
[0;34m[INFO][0m Restoring synchronous replication settings...
t
[0;34m[INFO][0m Worker Citus extension installed - registration will happen when coordinator setup runs
[0;32m[OK][0m Citus setup complete for worker-01
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m✓[0m ✅ Citus extension installed on worker-01
[0;32m✓[0m ✅ Phase 1 Complete: All 1 workers have Citus extension installed
[0;34m[INFO][0m 🔧 PHASE 2: Setting up Citus coordinator and registering workers...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS CLUSTER SETUP
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m 🔧 Setting up Citus Coordinator...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m DIAGNOSTIC: Configuration Variables
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m PG_WORKERS_NUM: 1
[0;34m[INFO][0m ENV_ID: web-universe-main-dev
[0;34m[INFO][0m DOMAIN: fastorder.com
[0;34m[INFO][0m PORT: 5432
[0;34m[INFO][0m SOCKET_DIR: /var/run/postgresql-web-universe-main-dev-coordinator
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Ensuring postgres client certificates exist for coordinator...
[0;32m[OK][0m Postgres client certificates already exist for coordinator
[0;34m[INFO][0m Adding citus_cert_map to coordinator pg_ident.conf...
[0;32m[OK][0m pg_ident.conf updated for coordinator
[0;34m[INFO][0m Installing Citus extension on coordinator...
[0;32m[OK][0m Citus extension installed on coordinator (postgres database)
[0;34m[INFO][0m Installing Citus extension on application database: fastorder_web_universe_main_dev_db...
[0;32m[OK][0m Citus extension installed on application database: fastorder_web_universe_main_dev_db
[0;34m[INFO][0m Configuring Citus SSL connection parameters...
[2026-02-05 07:21:16 UTC] USER=www-data EUID=0 PID=173181 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m ✅ Citus SSL connection parameters configured: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[1;33m[WARN][0m Node not identified as coordinator, initializing...
[0;34m[INFO][0m Checking coordinator configuration...
[0;34m[INFO][0m Persisting citus.local_hostname to postgresql.conf...
[2026-02-05 07:21:18 UTC] USER=www-data EUID=0 PID=173257 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/web-universe-main-dev/coordinator/postgresql.conf
[2026-02-05 07:21:19 UTC] USER=www-data EUID=0 PID=173289 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m ✅ citus.local_hostname persisted to config and reloaded
[0;34m[INFO][0m Configuring coordinator hostname in postgres database: db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432
[0;32m[OK][0m ✅ Coordinator hostname set to db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432 in postgres database
[0;34m[INFO][0m Checking coordinator configuration in application database: fastorder_web_universe_main_dev_db...
[0;34m[INFO][0m Configuring coordinator hostname in application database: db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432
[0;32m[OK][0m ✅ Coordinator hostname set to db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432 in application database
[0;34m[INFO][0m Validating coordinator configuration before worker registration...
[0;32m[OK][0m ✅ Coordinator hostname validated: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[0;32m[OK][0m ✅ citus_tables view is accessible
[0;34m[INFO][0m Checking coordinator self-registration...
[0;32m[OK][0m ✅ Coordinator is already self-registered
[0;34m[INFO][0m Configuring coordinator shard placement policy...
[0;32m[OK][0m ✅ Coordinator already configured in postgres database (shouldhaveshards = false)
[0;32m[OK][0m ✅ Coordinator already configured in application database (shouldhaveshards = false)
[0;34m[INFO][0m Registering 1 worker(s) to Citus cluster...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m PRE-FLIGHT: Checking worker availability...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Checking worker worker-01...
[0;34m[INFO][0m FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[0;32m[OK][0m ✅ Worker worker-01 is reachable via SSL
[0;32m[OK][0m All workers are reachable - proceeding with registration
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Adding Citus worker: db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Adding citus_cert_map to worker-01 pg_ident.conf...
[0;32m[OK][0m pg_ident.conf updated for worker-01
[0;34m[INFO][0m Configuring worker worker-01 HBA for coordinator (10.100.1.54) access...
[0;32m[OK][0m Worker worker-01 HBA configured for coordinator (10.100.1.54)
[0;34m[INFO][0m Adding replication rules for 3 standby(s)...
[0;32m[OK][0m Replication rules added for worker-01
[0;34m[INFO][0m Reloading worker worker-01 to apply HBA changes...
[2026-02-05 07:21:22 UTC] USER=www-data EUID=0 PID=173504 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-worker-01.service
[0;34m[INFO][0m Configuring coordinator HBA for worker worker-01 (10.100.1.243) access...
[0;32m[OK][0m Coordinator HBA configured for worker worker-01 (10.100.1.243)
[0;34m[INFO][0m Reloading coordinator to apply HBA changes...
[2026-02-05 07:21:22 UTC] USER=www-data EUID=0 PID=173549 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
[0;34m[INFO][0m Ensuring postgres client certificates exist for worker-01...
[0;32m[OK][0m Postgres client certificates already exist for worker-01
[0;34m[INFO][0m Configuring citus.node_conninfo on worker-01...
[2026-02-05 07:21:22 UTC] USER=www-data EUID=0 PID=173566 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-worker-01.service
[0;32m[OK][0m citus.node_conninfo configured on worker-01
[0;34m[INFO][0m Temporarily relaxing sync-rep on worker worker-01...
t
[0;32m[OK][0m Worker worker-01 sync-rep relaxed (was: sync_commit=on)
[0;34m[INFO][0m Ensuring Citus extension on worker databases...
CREATE EXTENSION
CREATE EXTENSION
[0;34m[INFO][0m Running citus_add_node with 180s timeout...
2
[0;34m[INFO][0m Restoring worker worker-01 sync-rep settings...
t
[0;32m[OK][0m Worker worker-01 sync-rep restored
[0;32m[OK][0m ✅ Worker db-web-universe-main-dev-postgresql-worker-01.fastorder.com successfully added to Citus cluster
[0;34m[INFO][0m Node ID: 2
[0;34m[INFO][0m Registered in: postgres, fastorder_web_universe_main_dev_db
[0;32m[OK][0m Worker worker-01 registration successful
[0;34m[INFO][0m Configuring worker worker-01 shard placement policy...
[0;32m[OK][0m ✅ Worker worker-01 configured to hold shards in all databases
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m POST-REGISTRATION: Verifying cluster state...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Expected workers: 1
[0;34m[INFO][0m Registered workers: 1
[0;32m[OK][0m ✅ All 1 workers successfully registered!
[0;34m[INFO][0m Citus cluster configuration:
db-web-universe-main-dev-postgresql-coordinator.fastorder.com 5432 0 t primary f
db-web-universe-main-dev-postgresql-worker-01.fastorder.com 5432 1 t primary t
[0;34m[INFO][0m Note: groupid=0 is the coordinator, groupid>0 are workers
[0;34m[INFO][0m shouldhaveshards: false=query router only, true=holds data shards
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m FINAL VALIDATION: Verifying configuration persistence...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:21:26 UTC] USER=www-data EUID=0 PID=173820 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/web-universe-main-dev/coordinator/postgresql.conf
[0;32m[OK][0m ✅ citus.local_hostname persisted in postgresql.conf
[0;32m[OK][0m ✅ All 1 worker(s) successfully registered and verified
[0;32m[OK][0m ✅ All validation checks passed
[0;32m[OK][0m Citus coordinator setup complete
[0;32m[OK][0m Citus setup complete for coordinator
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m✓[0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✓[0m ✅ CITUS CLUSTER SETUP COMPLETED SUCCESSFULLY
[0;32m✓[0m Coordinator: Ready and accepting connections
[0;32m✓[0m Workers registered: 1
[0;32m✓[0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 05-backup-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up coordinator backup...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for web-universe-main-dev...
[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO] Version: pgBackRest 2.56.0
[INFO] 2️⃣ Creating backup directories...
[2026-02-05 07:21:28 UTC] USER=www-data EUID=0 PID=173932 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 07:21:28 UTC] USER=www-data EUID=0 PID=173941 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/web-universe-main-dev
[2026-02-05 07:21:28 UTC] USER=www-data EUID=0 PID=173950 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-02-05 07:21:28 UTC] USER=www-data EUID=0 PID=173959 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-02-05 07:21:28 UTC] USER=www-data EUID=0 PID=173968 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-02-05 07:21:28 UTC] USER=www-data EUID=0 PID=173977 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-02-05 07:21:30 UTC] USER=www-data EUID=0 PID=174077 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-02-05 07:21:30 UTC] USER=www-data EUID=0 PID=174086 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-02-05 07:21:30 UTC] USER=www-data EUID=0 PID=174095 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-02-05 07:21:30 UTC] USER=www-data EUID=0 PID=174104 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 07:21:30 UTC] USER=www-data EUID=0 PID=174115 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO] ✅ Backup directories created
[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-web-universe-main-dev
[2026-02-05 07:21:30 UTC] USER=www-data EUID=0 PID=174145 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key
[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-02-05 07:21:30 UTC] USER=www-data EUID=0 PID=174156 ACTION=fsop ARGS=find /data/postgresql/17/web-universe-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-02-05 07:21:30 UTC] USER=www-data EUID=0 PID=174173 ACTION=fsop ARGS=chown -R postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed
[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-02-05 07:21:30 UTC] USER=www-data EUID=0 PID=174182 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-02-05 07:21:30 UTC] USER=www-data EUID=0 PID=174191 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-02-05 07:21:31 UTC] USER=www-data EUID=0 PID=174202 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created
[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-02-05 07:21:31 UTC] USER=www-data EUID=0 PID=174211 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/coordinator/PG_VERSION
[2026-02-05 07:21:31 UTC] USER=www-data EUID=0 PID=174232 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running
[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza web-universe-main-dev-coordinator already initialized and verified
[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
pg_reload_conf
----------------
t
(1 row)
[INFO] ✅ WAL archiving configured for coordinator
[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-02-05 07:21:31 UTC] USER=www-data EUID=0 PID=174288 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-02-05 07:21:33 UTC] USER=www-data EUID=0 PID=174349 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 07:21:37 UTC] USER=www-data EUID=0 PID=174525 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled
[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-02-05 07:21:38 UTC] USER=www-data EUID=0 PID=174549 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator --log-level-console=info check
2026-02-05 07:21:38.036 P00 INFO: check command begin 2.56.0: --exec-id=174556-753fd729 --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 07:21:38.061 P00 INFO: check repo1 configuration (primary)
2026-02-05 07:21:38.072 P00 ERROR: [028]: backup and archive info files exist but do not match the database
HINT: is this the correct stanza?
HINT: did an error occur during stanza-upgrade?
2026-02-05 07:21:38.072 P00 INFO: check command end: aborted with exception [028]
[WARN] ⚠️ Stanza verification failed - this may be normal if WAL archiving hasn't started yet
[WARN] The backup system is configured and will work once WAL segments are generated
[INFO] 8️⃣ Creating backup automation scripts...
[2026-02-05 07:21:38 UTC] USER=www-data EUID=0 PID=174571 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 07:21:38 UTC] USER=www-data EUID=0 PID=174580 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 07:21:38 UTC] USER=www-data EUID=0 PID=174598 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[2026-02-05 07:21:38 UTC] USER=www-data EUID=0 PID=174608 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[INFO] ✅ Backup scripts created
[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-02-05 07:21:38 UTC] USER=www-data EUID=0 PID=174628 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-web-universe-main-dev
[INFO] ✅ Cron jobs configured
[INFO] Schedule:
[INFO] - Full backup: Sundays at 2:00 AM
[INFO] - Differential backup: Mon-Sat at 2:00 AM
[INFO] 🔟 Creating restore documentation...
[2026-02-05 07:21:38 UTC] USER=www-data EUID=0 PID=174646 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 07:21:38 UTC] USER=www-data EUID=0 PID=174655 ACTION=fsop ARGS=sed -i s|__ENV_ID__|web-universe-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 07:21:38 UTC] USER=www-data EUID=0 PID=174664 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/data/postgresql/17/web-universe-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 07:21:38 UTC] USER=www-data EUID=0 PID=174673 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 07:21:38 UTC] USER=www-data EUID=0 PID=174682 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-02-05 07:21:38.583 P00 INFO: start command begin 2.56.0: --exec-id=174704-d3d845c8 --log-level-console=info --log-level-file=debug --stanza=web-universe-main-dev-coordinator
2026-02-05 07:21:38.583 P00 WARN: stop file does not exist for stanza web-universe-main-dev-coordinator
2026-02-05 07:21:38.583 P00 INFO: start command end: completed successfully (3ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-02-05 07:21:38.663 P00 INFO: stanza-upgrade command begin 2.56.0: --exec-id=174724-6599facd --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 07:21:38.664 P00 INFO: stanza-upgrade for stanza 'web-universe-main-dev-coordinator' on repo1
2026-02-05 07:21:38.675 P00 INFO: stanza-upgrade command end: completed successfully (15ms)
[INFO] This may take a few minutes depending on database size...
[2026-02-05 07:21:38 UTC] USER=www-data EUID=0 PID=174732 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260205-072138.log
[2026-02-05 07:21:38 UTC] USER=www-data EUID=0 PID=174741 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260205-072138.log
[2026-02-05 07:21:38 UTC] USER=www-data EUID=0 PID=174750 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260205-072138.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-02-05 07:21:52 UTC] USER=www-data EUID=0 PID=175286 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-173906.log /var/log/pgbackrest/initial-backup-20260205-072138.log
[INFO] ✅ Initial full backup completed successfully
[INFO] Log: /var/log/pgbackrest/initial-backup-20260205-072138.log
2026-02-05 07:21:52.424 P00 INFO: expire command begin 2.56.0: --exec-id=174768-6f7d21ef --log-level-console=info --log-level-file=debug --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --repo1-retention-diff=4 --repo1-retention-full=4 --stanza=web-universe-main-dev-coordinator
2026-02-05 07:21:52.425 P00 INFO: repo1: 17-1 remove archive, start = 000000010000000000000003, stop = 000000010000000000000004
2026-02-05 07:21:52.426 P00 INFO: repo1: 17-2 remove archive, start = 000000010000000000000002, stop = 000000010000000000000002
2026-02-05 07:21:52.426 P00 INFO: repo1: 17-3 remove archive, start = 000000010000000000000002, stop = 000000010000000000000002
2026-02-05 07:21:52.426 P00 INFO: expire command end: completed successfully (2ms)
[INFO] Current backups:
stanza: web-universe-main-dev-coordinator
status: ok
cipher: aes-256-cbc
db (prior)
wal archive min/max (17): 000000010000000000000005/000000010000000000000008
full backup: 20260205-065853F
timestamp start/stop: 2026-02-05 06:58:53+00 / 2026-02-05 06:58:56+00
wal start/stop: 000000010000000000000005 / 000000010000000000000005
database size: 33.5MB, database backup size: 33.5MB
repo1: backup set size: 5.4MB, backup size: 5.4MB
db (prior)
wal archive min/max (17): 000000010000000000000003/000000010000000000000007
full backup: 20260205-071517F
timestamp start/stop: 2026-02-05 07:15:17+00 / 2026-02-05 07:15:26+00
wal start/stop: 000000010000000000000003 / 000000010000000000000003
database size: 33.5MB, database backup size: 33.5MB
repo1: backup set size: 5.4MB, backup size: 5.4MB
full backup: 20260205-071539F
timestamp start/stop: 2026-02-05 07:15:39+00 / 2026-02-05 07:15:41+00
wal start/stop: 000000010000000000000006 / 000000010000000000000006
database size: 33.5MB, database backup size: 33.5MB
repo1: backup set size: 5.4MB, backup size: 5.4MB
db (current)
wal archive min/max (17): none present
full backup: 20260205-072138F
timestamp start/stop: 2026-02-05 07:21:38+00 / 2026-02-05 07:21:52+00
wal start/stop: 000000010000000000000003 / 000000010000000000000003
database size: 33.5MB, database backup size: 33.5MB
repo1: backup set size: 5.4MB, backup size: 5.4MB
[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️ No worker identifier provided - skipping worker backup setup
[INFO] (Run with 'worker-01', 'worker-02', etc. to configure worker backups)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Completed steps:
[INFO] 1. pgBackRest installed and configured
[INFO] 2. WAL archiving enabled (archive_mode=on)
[INFO] 3. PostgreSQL restarted with new settings
[INFO] 4. pgBackRest stanza initialized and verified
[INFO] 5. Initial full backup completed
[INFO] 6. Automated backup cron jobs configured
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO] Coordinator:
[INFO] Stanza: web-universe-main-dev-coordinator
[INFO] Schedule: Full: Sun 2AM, Diff: Mon-Sat 2AM
[INFO] Common:
[INFO] Backup dir: /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO] Archive dir: /var/lib/pgbackrest/archive/web-universe-main-dev
[INFO] Config: /etc/pgbackrest/pgbackrest.conf
[INFO] Restore guide: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] Retention:
[INFO] Full backups: 4 (keep last 4 full backups)
[INFO] Differential: 4 (keep last 4 diff per full)
[INFO] Archive WAL: Auto-managed by pgBackRest
[INFO] Manual commands:
[INFO] Coordinator: sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator backup
[INFO] List all backups: sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up worker backups for 1 worker(s)...
[0;34m[INFO][0m Setting up backup for: worker-01
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for web-universe-main-dev...
[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO] Version: pgBackRest 2.56.0
[INFO] 2️⃣ Creating backup directories...
[2026-02-05 07:21:52 UTC] USER=www-data EUID=0 PID=175339 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 07:21:52 UTC] USER=www-data EUID=0 PID=175348 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/web-universe-main-dev
[2026-02-05 07:21:52 UTC] USER=www-data EUID=0 PID=175357 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-02-05 07:21:52 UTC] USER=www-data EUID=0 PID=175366 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-02-05 07:21:52 UTC] USER=www-data EUID=0 PID=175377 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-02-05 07:21:53 UTC] USER=www-data EUID=0 PID=175387 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-02-05 07:21:54 UTC] USER=www-data EUID=0 PID=175493 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-02-05 07:21:54 UTC] USER=www-data EUID=0 PID=175513 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-02-05 07:21:54 UTC] USER=www-data EUID=0 PID=175524 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-02-05 07:21:55 UTC] USER=www-data EUID=0 PID=175540 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 07:21:55 UTC] USER=www-data EUID=0 PID=175549 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO] ✅ Backup directories created
[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-web-universe-main-dev
[2026-02-05 07:21:55 UTC] USER=www-data EUID=0 PID=175570 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-02-05 07:21:55 UTC] USER=www-data EUID=0 PID=175579 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key
[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-02-05 07:21:55 UTC] USER=www-data EUID=0 PID=175588 ACTION=fsop ARGS=find /data/postgresql/17/web-universe-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-02-05 07:21:55 UTC] USER=www-data EUID=0 PID=175597 ACTION=fsop ARGS=chown -R postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed
[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-02-05 07:21:55 UTC] USER=www-data EUID=0 PID=175607 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-02-05 07:21:55 UTC] USER=www-data EUID=0 PID=175616 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-02-05 07:21:55 UTC] USER=www-data EUID=0 PID=175625 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created
[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-02-05 07:21:55 UTC] USER=www-data EUID=0 PID=175634 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/coordinator/PG_VERSION
[2026-02-05 07:21:55 UTC] USER=www-data EUID=0 PID=175644 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running
[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza web-universe-main-dev-coordinator already initialized and verified
[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
pg_reload_conf
----------------
t
(1 row)
[INFO] ✅ WAL archiving configured for coordinator
[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-02-05 07:21:56 UTC] USER=www-data EUID=0 PID=175716 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-02-05 07:21:58 UTC] USER=www-data EUID=0 PID=175841 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 07:22:02 UTC] USER=www-data EUID=0 PID=176017 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled
[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-02-05 07:22:02 UTC] USER=www-data EUID=0 PID=176044 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator --log-level-console=info check
2026-02-05 07:22:02.452 P00 INFO: check command begin 2.56.0: --exec-id=176052-c85f64e0 --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 07:22:02.471 P00 INFO: check repo1 configuration (primary)
2026-02-05 07:22:02.516 P00 INFO: check repo1 archive for WAL (primary)
2026-02-05 07:22:02.817 P00 INFO: WAL segment 000000010000000000000005 successfully archived to '/var/lib/pgbackrest/backup/web-universe-main-dev/archive/web-universe-main-dev-coordinator/17-3/0000000100000000/000000010000000000000005-d77d3cc55133d8beb92d0878b81a4d4820efc3d9.lz4' on repo1
2026-02-05 07:22:02.817 P00 INFO: check command end: completed successfully (371ms)
[INFO] ✅ Stanza verification passed
[INFO] 8️⃣ Creating backup automation scripts...
[2026-02-05 07:22:02 UTC] USER=www-data EUID=0 PID=176081 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 07:22:02 UTC] USER=www-data EUID=0 PID=176090 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 07:22:03 UTC] USER=www-data EUID=0 PID=176112 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[2026-02-05 07:22:03 UTC] USER=www-data EUID=0 PID=176121 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[INFO] ✅ Backup scripts created
[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-02-05 07:22:03 UTC] USER=www-data EUID=0 PID=176139 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-web-universe-main-dev
[INFO] ✅ Cron jobs configured
[INFO] Schedule:
[INFO] - Full backup: Sundays at 2:00 AM
[INFO] - Differential backup: Mon-Sat at 2:00 AM
[INFO] 🔟 Creating restore documentation...
[2026-02-05 07:22:03 UTC] USER=www-data EUID=0 PID=176158 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 07:22:03 UTC] USER=www-data EUID=0 PID=176169 ACTION=fsop ARGS=sed -i s|__ENV_ID__|web-universe-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 07:22:03 UTC] USER=www-data EUID=0 PID=176178 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/data/postgresql/17/web-universe-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 07:22:03 UTC] USER=www-data EUID=0 PID=176188 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 07:22:03 UTC] USER=www-data EUID=0 PID=176197 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-02-05 07:22:03.789 P00 INFO: start command begin 2.56.0: --exec-id=176219-3ce0d458 --log-level-console=info --log-level-file=debug --stanza=web-universe-main-dev-coordinator
2026-02-05 07:22:03.790 P00 WARN: stop file does not exist for stanza web-universe-main-dev-coordinator
2026-02-05 07:22:03.790 P00 INFO: start command end: completed successfully (10ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-02-05 07:22:03.870 P00 INFO: stanza-upgrade command begin 2.56.0: --exec-id=176231-1ae17e8e --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 07:22:03.871 P00 INFO: stanza-upgrade for stanza 'web-universe-main-dev-coordinator' on repo1
2026-02-05 07:22:03.872 P00 INFO: stanza 'web-universe-main-dev-coordinator' on repo1 is already up to date
2026-02-05 07:22:03.872 P00 INFO: stanza-upgrade command end: completed successfully (9ms)
[INFO] This may take a few minutes depending on database size...
[2026-02-05 07:22:03 UTC] USER=www-data EUID=0 PID=176236 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260205-072203.log
[2026-02-05 07:22:03 UTC] USER=www-data EUID=0 PID=176250 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260205-072203.log
[2026-02-05 07:22:04 UTC] USER=www-data EUID=0 PID=176262 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260205-072203.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-02-05 07:22:06 UTC] USER=www-data EUID=0 PID=176381 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-175313.log /var/log/pgbackrest/initial-backup-20260205-072203.log
[INFO] ✅ Initial full backup completed successfully
[INFO] Log: /var/log/pgbackrest/initial-backup-20260205-072203.log
2026-02-05 07:22:06.814 P00 INFO: repo1: remove expired backup 20260205-065853F
2026-02-05 07:22:06.845 P00 INFO: repo1: remove archive path /var/lib/pgbackrest/backup/web-universe-main-dev/archive/web-universe-main-dev-coordinator/17-1
2026-02-05 07:22:06.845 P00 INFO: repo1: 17-2 no archive to remove
2026-02-05 07:22:06.846 P00 INFO: repo1: 17-3 no archive to remove
2026-02-05 07:22:06.846 P00 INFO: expire command end: completed successfully (37ms)
[INFO] Current backups:
stanza: web-universe-main-dev-coordinator
status: ok
cipher: aes-256-cbc
db (prior)
wal archive min/max (17): 000000010000000000000003/000000010000000000000007
full backup: 20260205-071517F
timestamp start/stop: 2026-02-05 07:15:17+00 / 2026-02-05 07:15:26+00
wal start/stop: 000000010000000000000003 / 000000010000000000000003
database size: 33.5MB, database backup size: 33.5MB
repo1: backup set size: 5.4MB, backup size: 5.4MB
full backup: 20260205-071539F
timestamp start/stop: 2026-02-05 07:15:39+00 / 2026-02-05 07:15:41+00
wal start/stop: 000000010000000000000006 / 000000010000000000000006
database size: 33.5MB, database backup size: 33.5MB
repo1: backup set size: 5.4MB, backup size: 5.4MB
db (current)
wal archive min/max (17): 000000010000000000000003/000000010000000000000006
full backup: 20260205-072138F
timestamp start/stop: 2026-02-05 07:21:38+00 / 2026-02-05 07:21:52+00
wal start/stop: 000000010000000000000003 / 000000010000000000000003
database size: 33.5MB, database backup size: 33.5MB
repo1: backup set size: 5.4MB, backup size: 5.4MB
full backup: 20260205-072204F
timestamp start/stop: 2026-02-05 07:22:04+00 / 2026-02-05 07:22:06+00
wal start/stop: 000000010000000000000006 / 000000010000000000000006
database size: 33.5MB, database backup size: 33.5MB
repo1: backup set size: 5.4MB, backup size: 5.4MB
[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️ No worker identifier provided - skipping worker backup setup
[INFO] (Run with 'worker-01', 'worker-02', etc. to configure worker backups)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Completed steps:
[INFO] 1. pgBackRest installed and configured
[INFO] 2. WAL archiving enabled (archive_mode=on)
[INFO] 3. PostgreSQL restarted with new settings
[INFO] 4. pgBackRest stanza initialized and verified
[INFO] 5. Initial full backup completed
[INFO] 6. Automated backup cron jobs configured
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO] Coordinator:
[INFO] Stanza: web-universe-main-dev-coordinator
[INFO] Schedule: Full: Sun 2AM, Diff: Mon-Sat 2AM
[INFO] Common:
[INFO] Backup dir: /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO] Archive dir: /var/lib/pgbackrest/archive/web-universe-main-dev
[INFO] Config: /etc/pgbackrest/pgbackrest.conf
[INFO] Restore guide: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] Retention:
[INFO] Full backups: 4 (keep last 4 full backups)
[INFO] Differential: 4 (keep last 4 diff per full)
[INFO] Archive WAL: Auto-managed by pgBackRest
[INFO] Manual commands:
[INFO] Coordinator: sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator backup
[INFO] List all backups: sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✓[0m ✅ Backup setup completed for coordinator and all workers
[0;34m[INFO][0m Skipping 06-distribute-tables-canary.sh (test script - set RUN_TESTS=true to enable)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 07-distribute-tables.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:22:08 UTC] USER=unknown EUID=33 PID=176471 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 07:22:08 UTC] USER=unknown EUID=33 PID=176478 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 07:22:08 UTC] USER=unknown EUID=33 PID=176485 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 07:22:08 UTC] USER=unknown EUID=33 PID=176492 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS TABLE DISTRIBUTION
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m 🔐 Secure connection established
[0;34m[INFO][0m Host: db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432
[0;34m[INFO][0m Database: fastorder_web_universe_main_dev_db
[0;34m[INFO][0m SSL: verify-full (TLS 1.2+)
[0;34m[INFO][0m Timeouts: statement=120s, idle_tx=300s
[0;34m[INFO][0m 🔍 Running preflight checks...
[0;34m[INFO][0m Testing database connectivity...
[0;32m[OK][0m ✅ Database connection successful
[0;32m[OK][0m ✅ Connected to correct database: fastorder_web_universe_main_dev_db
[0;34m[INFO][0m Checking Citus extension in database fastorder_web_universe_main_dev_db...
[0;32m[OK][0m Citus version: 13.2-1
[0;34m[INFO][0m Checking worker registration...
[0;32m[OK][0m Registered workers: 1
[0;34m[INFO][0m Worker nodes:
[0;34m[INFO][0m nodename | nodeport | isactive | noderole
[0;34m[INFO][0m -------------------------------------------------------------+----------+----------+----------
[0;34m[INFO][0m db-web-universe-main-dev-postgresql-worker-01.fastorder.com | 5432 | t | primary
[0;34m[INFO][0m (1 row)
[0;34m[INFO][0m
[0;34m[INFO][0m 📊 Starting table distribution...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Distributing: auth.login_account
[0;34m[INFO][0m Description: User authentication table - distributed by region for tenant isolation
[0;34m[INFO][0m Shard key: region_hint
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ⏭️ Table does not exist, skipping
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ All tables distributed successfully!
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m 📊 Citus Cluster Summary:
[0;34m[INFO][0m Distributed tables:
[0;34m[INFO][0m table | type | shard_key | shards | size
[0;34m[INFO][0m -------+------+-----------+--------+------
[0;34m[INFO][0m (0 rows)
[0;34m[INFO][0m
[0;34m[INFO][0m Worker capacity:
[0;34m[INFO][0m worker | total_shards | total_size
[0;34m[INFO][0m --------+--------------+------------
[0;34m[INFO][0m (0 rows)
[0;34m[INFO][0m
[0;32m[OK][0m Citus table distribution complete
[0;34m[INFO][0m Skipping 08-distribute-tables-rollback.sh (rollback script - run manually only)
[0;34m[INFO][0m Skipping 09-distribute-tables-test.sh (test script - set RUN_TESTS=true to enable)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 10-setup-cdc.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m CDC PIPELINE SETUP (Debezium + Elasticsearch Sink)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Log file: /var/log/fastorder/cdc/10-setup-cdc-*.log
[0;34m[INFO][0m Running CDC setup for identifier: coordinator
[2026-02-05 07:22:13] ==========================================
[2026-02-05 07:22:13] CDC SETUP SCRIPT STARTED
[2026-02-05 07:22:13] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260205_072213.log
[2026-02-05 07:22:13] ==========================================
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 07:22:13] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:22:13] CDC Pipeline Setup (Debezium + ES Sink)
[2026-02-05 07:22:13] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:22:13] Environment: web-universe-main-dev
[2026-02-05 07:22:13] Identifier: coordinator
[2026-02-05 07:22:13] Service: web
[2026-02-05 07:22:13] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:22:13] 📂 CDC_BASE_DIR exists: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc
[2026-02-05 07:22:13] Looking for service folder: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web
[2026-02-05 07:22:13]
[2026-02-05 07:22:13] 📂 Found CDC configuration for service: web
[2026-02-05 07:22:13] Scanning for subservice directories in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web
[2026-02-05 07:22:13] Found subservice: config, checking for steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps
[2026-02-05 07:22:13]
[2026-02-05 07:22:13] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:22:13] Setting up CDC for: web/config
[2026-02-05 07:22:13] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:22:13] Found 3 step script(s) in /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps
[2026-02-05 07:22:13]
[2026-02-05 07:22:13] 🔧 Running: 01-setup-config-cdc.sh
[2026-02-05 07:22:13] Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps/01-setup-config-cdc.sh
[2026-02-05 07:22:13] Executing directly (script is executable)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Config CDC Setup - Automatic Role Detection
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service: web
[INFO] Zone: universe
[INFO] Branch: main
[INFO] Environment: dev
[INFO] Identifier: coordinator
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service 'web' requires config.* schema
[INFO] CDC Role for web in zone universe: master
[INFO] ════════════════════════════════════════════════════════════
[INFO] CONTROL PLANE MASTER (zone=universe)
[INFO] Setting up Debezium CDC Publisher
[INFO] ════════════════════════════════════════════════════════════
[INFO] Executing Debezium config setup...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Debezium Config CDC Master Setup
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: web-universe-main-dev
[INFO] PostgreSQL Host: db-web-universe-main-dev-postgresql.fastorder.com
[INFO] Database: fastorder_web_universe_main_dev_db
[INFO] Connector: debezium-web-universe-main-dev-config
[INFO] Kafka Connect: eventbus-web-universe-main-dev-kafka-connect.fastorder.com:8083
[INFO] SSL Cert Dir: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator (on Kafka Connect host)
[INFO] SSL Key File: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der
[INFO] Topic Prefix: cdc.web_universe_main_dev
[INFO] Replication Slot: dbz_web_universe_main_dev_config
[INFO] Tables: config.public_defaults,config.feature_flags,config.config_version
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Step 1: Creating PostgreSQL publication for config tables...
[INFO] Creating publication: cdc_pub_web_config
CREATE PUBLICATION
[INFO] Setting REPLICA IDENTITY FULL for config tables...
ALTER TABLE
ALTER TABLE
ALTER TABLE
[INFO] Step 2: Creating replication slot...
[INFO] Creating replication slot: dbz_web_universe_main_dev_config
(dbz_web_universe_main_dev_config,0/700E210)
[INFO] Step 3: Registering Debezium connector with Kafka Connect...
[INFO] Connector debezium-web-universe-main-dev-config already exists - updating configuration
[INFO] Sending connector configuration to Kafka Connect...
[ OK ] Debezium connector registered successfully
[INFO] Step 4: Verifying connector status...
[INFO] Connector State: RUNNING
[INFO] Task State: RUNNING
[ OK ] Debezium connector is running
[INFO]
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Debezium Config CDC Master Setup Complete
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Connector: debezium-web-universe-main-dev-config
[INFO] Topic Prefix: cdc.web_universe_main_dev
[INFO] Publication: cdc_pub_web_config
[INFO] Replication Slot: dbz_web_universe_main_dev_config
[INFO]
[INFO] Topics Created:
[INFO] - cdc.web_universe_main_dev.config.public_defaults
[INFO] - cdc.web_universe_main_dev.config.feature_flags
[INFO] - cdc.web_universe_main_dev.config.config_version
[INFO]
[INFO] Data Planes (replicas) should subscribe to:
[INFO] Topic Prefix: cdc.web_universe_main_dev
[INFO] ═══════════════════════════════════════════════════════════════
[ OK ] Debezium config CDC master setup complete
[ OK ] Debezium config CDC master setup complete
[INFO] No topology.json found at /opt/fastorder/state/web-universe-main-dev/topology.json - skipping merge
[INFO]
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Config CDC Setup Complete
[INFO] ═══════════════════════════════════════════════════════════════
═══════════════════════════════════════════════════════════════
Service Capabilities: web-universe-main-dev
═══════════════════════════════════════════════════════════════
Service: web
Zone: universe
Branch: main
Environment: dev
Config Schema: ✅ YES
Redis Cache: ✅ YES
CDC Role: master
CDC Master Configuration:
Debezium: debezium-web-universe-main-dev-config
Topic Prefix: cdc.web_universe_main_dev
Repl Slot: dbz_web_universe_main_dev_config
Tables: config.public_defaults,config.feature_flags,config.config_version
Required Schemas: config tenant dashboard environment resource service item company communication ai
═══════════════════════════════════════════════════════════════
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]
[INFO] Log file: /var/log/fastorder/cdc/config-cdc-20260205_072213.log
[ OK ] Config CDC setup finished successfully
[2026-02-05 07:22:21] ✅ Completed: 01-setup-config-cdc.sh
[2026-02-05 07:22:21]
[2026-02-05 07:22:21] 🔧 Running: 02-setup-debezium-config.sh
[2026-02-05 07:22:21] Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps/02-setup-debezium-config.sh
[2026-02-05 07:22:21] Executing directly (script is executable)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Debezium Config CDC Master Setup
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: web-universe-main-dev
[INFO] PostgreSQL Host: db-web-universe-main-dev-postgresql.fastorder.com
[INFO] Database: fastorder_web_universe_main_dev_db
[INFO] Connector: debezium-web-universe-main-dev-config
[INFO] Kafka Connect: eventbus-web-universe-main-dev-kafka-connect.fastorder.com:8083
[INFO] SSL Cert Dir: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator (on Kafka Connect host)
[INFO] SSL Key File: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der
[INFO] Topic Prefix: cdc.web_universe_main_dev
[INFO] Replication Slot: dbz_web_universe_main_dev_config
[INFO] Tables: config.public_defaults,config.feature_flags,config.config_version
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Step 1: Creating PostgreSQL publication for config tables...
[INFO] Publication cdc_pub_web_config already exists
[INFO] Setting REPLICA IDENTITY FULL for config tables...
ALTER TABLE
ALTER TABLE
ALTER TABLE
[INFO] Step 2: Creating replication slot...
[INFO] Replication slot dbz_web_universe_main_dev_config already exists
[INFO] Step 3: Registering Debezium connector with Kafka Connect...
[INFO] Connector debezium-web-universe-main-dev-config already exists - updating configuration
[INFO] Sending connector configuration to Kafka Connect...
[ OK ] Debezium connector registered successfully
[INFO] Step 4: Verifying connector status...
[INFO] Connector State: RUNNING
[INFO] Task State: RUNNING
[ OK ] Debezium connector is running
[INFO]
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Debezium Config CDC Master Setup Complete
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Connector: debezium-web-universe-main-dev-config
[INFO] Topic Prefix: cdc.web_universe_main_dev
[INFO] Publication: cdc_pub_web_config
[INFO] Replication Slot: dbz_web_universe_main_dev_config
[INFO]
[INFO] Topics Created:
[INFO] - cdc.web_universe_main_dev.config.public_defaults
[INFO] - cdc.web_universe_main_dev.config.feature_flags
[INFO] - cdc.web_universe_main_dev.config.config_version
[INFO]
[INFO] Data Planes (replicas) should subscribe to:
[INFO] Topic Prefix: cdc.web_universe_main_dev
[INFO] ═══════════════════════════════════════════════════════════════
[ OK ] Debezium config CDC master setup complete
[2026-02-05 07:22:28] ✅ Completed: 02-setup-debezium-config.sh
[2026-02-05 07:22:28]
[2026-02-05 07:22:28] 🔧 Running: 03-setup-kafka-consumer.sh
[2026-02-05 07:22:28] Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps/03-setup-kafka-consumer.sh
[2026-02-05 07:22:28] Executing directly (script is executable)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] Skipping Kafka consumer setup - Control Plane (universe) uses Debezium publisher
[INFO] Current zone: universe
[INFO] Debezium config was set up by 02-setup-debezium-config.sh
[2026-02-05 07:22:28] ✅ Completed: 03-setup-kafka-consumer.sh
[2026-02-05 07:22:28]
[2026-02-05 07:22:28] ==========================================
[2026-02-05 07:22:28] ✅ CDC Pipeline setup complete for 1 subservice(s)
[2026-02-05 07:22:28] CDC SETUP SCRIPT FINISHED
[2026-02-05 07:22:28] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260205_072213.log
[2026-02-05 07:22:28] ==========================================
[0;32m✓[0m ✅ CDC Pipeline setup completed
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 11-monitoring-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up monitoring for coordinator...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;32m✓ [SECRETS][0m Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[0;34m[SECRETS][0m Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[0;34m[SECRETS][0m Search (build_es_secret_name, get/set_es_credentials_to_vault)
[0;34m[SECRETS][0m Backups (build_backup_path)
[0;34m[SECRETS][0m Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 PostgreSQL Monitoring Integration for web-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-web-universe-main-dev
[1;32m[OK][0m Observability cell endpoints registered for web-universe-main-dev
[1;32m[OK][0m ✓ Observability cell is ready
[INFO] ✓ Using private IP for metrics: 10.100.1.243
[INFO] 2️⃣ Setting up postgres_exporter integration...
[INFO] Checking observability cell readiness: obs-web-universe-main-dev
[1;32m[OK][0m Observability cell endpoints registered for web-universe-main-dev
[INFO] Setting up postgres_exporter for web-universe-main-dev
[2026-02-05 07:22:30 UTC] USER=www-data EUID=0 PID=177773 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-web-universe-main-dev.yaml /etc/prometheus/postgres_exporter_queries-web-universe-main-dev.yaml
[2026-02-05 07:22:30 UTC] USER=www-data EUID=0 PID=177795 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-web-universe-main-dev.yaml
[2026-02-05 07:22:30 UTC] USER=www-data EUID=0 PID=177811 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-web-universe-main-dev.yaml
[1;32m[OK][0m Custom queries file created at /etc/prometheus/postgres_exporter_queries-web-universe-main-dev.yaml
[2026-02-05 07:22:30 UTC] USER=www-data EUID=0 PID=177841 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 07:22:31 UTC] USER=www-data EUID=0 PID=177913 ACTION=passthru ARGS=systemctl enable postgres_exporter-web-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/postgres_exporter-web-universe-main-dev.service → /etc/systemd/system/postgres_exporter-web-universe-main-dev.service.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
IP Conflict Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
IP Address: 10.100.1.54
Port: 9187
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔍 Checking IP conflict for web-universe-main-dev on 10.100.1.54:9187...
✅ IP 10.100.1.54:9187 is available - no conflicts detected
🔍 Checking for orphaned processes that might conflict...
✅ No orphaned processes detected
✅ All checks passed - safe to proceed with web-universe-main-dev setup
[2026-02-05 07:22:32 UTC] USER=www-data EUID=0 PID=178096 ACTION=passthru ARGS=systemctl restart postgres_exporter-web-universe-main-dev.service
[1;32m[OK][0m postgres_exporter configured on db-web-universe-main-dev-postgresql.fastorder.com:9187
[INFO] Adding PostgreSQL scrape target to Prometheus config...
[1;32m[OK][0m PostgreSQL scrape target added
[INFO] Creating PostgreSQL alert rules...
[2026-02-05 07:22:34 UTC] USER=www-data EUID=0 PID=178181 ACTION=fsop ARGS=mv /tmp/postgresql_alerts_web-universe-main-dev.yml /etc/prometheus/obs-web-universe-main-dev/rules/postgresql_alerts.yml
[1;32m[OK][0m PostgreSQL alert rules created: /etc/prometheus/obs-web-universe-main-dev/rules/postgresql_alerts.yml
[INFO] Adding PostgreSQL alerts to Prometheus config...
[2026-02-05 07:22:34 UTC] USER=www-data EUID=0 PID=178192 ACTION=fsop ARGS=sed -i /rule_files:/a\ - "rules/postgresql_alerts.yml" /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
[1;32m[OK][0m PostgreSQL alerts registered in Prometheus
[2026-02-05 07:22:34 UTC] USER=www-data EUID=0 PID=178202 ACTION=passthru ARGS=systemctl reload prometheus-obs-web-universe-main-dev.service
Failed to reload prometheus-obs-web-universe-main-dev.service: Job type reload is not applicable for unit prometheus-obs-web-universe-main-dev.service.
[2026-02-05 07:22:34 UTC] USER=www-data EUID=0 PID=178211 ACTION=passthru ARGS=systemctl restart prometheus-obs-web-universe-main-dev.service
[1;32m[OK][0m Prometheus reloaded with PostgreSQL monitoring
[1;32m[OK][0m ✓ postgres_exporter integration complete
[INFO] Registering postgres_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: postgres_exporter -> 10.100.1.243:9187
[1;32m[OK][0m ✓ Registered postgres_exporter scrape target: 10.100.1.243:9187
[INFO] Target file: /etc/prometheus/obs-web-universe-main-dev/targets/postgres_exporter.yml
[1;32m[OK][0m ✓ postgres_exporter registered as Prometheus scrape target
[INFO] 3️⃣ Setting up pgbouncer_exporter integration...
[INFO] PgBouncer FQDN found in /etc/hosts: db-web-universe-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.244
[INFO] PgBouncer detected: db-web-universe-main-dev-postgresql-bouncer.fastorder.com:6432
[1;32m[OK][0m ✓ pgbouncer_exporter already installed
[INFO] Getting pgbouncer_admin password (SERVICE=web, ZONE=universe)
[1;32m[OK][0m ✓ pgbouncer_admin password retrieved (24 chars)
[INFO] Using pgbouncer certs from: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[INFO] Creating pgbouncer_exporter systemd service...
[1;32m[OK][0m ✓ pgbouncer_exporter service file created
[INFO] Starting pgbouncer_exporter service...
[2026-02-05 07:22:35 UTC] USER=www-data EUID=0 PID=178314 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 07:22:36 UTC] USER=www-data EUID=0 PID=178363 ACTION=passthru ARGS=systemctl enable pgbouncer_exporter-web-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/pgbouncer_exporter-web-universe-main-dev.service → /etc/systemd/system/pgbouncer_exporter-web-universe-main-dev.service.
[2026-02-05 07:22:36 UTC] USER=www-data EUID=0 PID=178429 ACTION=passthru ARGS=systemctl restart pgbouncer_exporter-web-universe-main-dev.service
[1;32m[OK][0m ✓ pgbouncer_exporter service running
[INFO] Registering pgbouncer_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: pgbouncer_exporter -> 10.100.1.244:9127
[1;32m[OK][0m ✓ Registered pgbouncer_exporter scrape target: 10.100.1.244:9127
[INFO] Target file: /etc/prometheus/obs-web-universe-main-dev/targets/pgbouncer_exporter.yml
[1;32m[OK][0m ✓ pgbouncer_exporter registered as Prometheus scrape target
[INFO] 4️⃣ Registering nodes to monitoring database...
[INFO] PostgreSQL key permissions set for www-data access: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[INFO] Registering PostgreSQL coordinator to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: PostgreSQL
[INFO] Identifier: web-universe-main-dev-postgresql-coordinator
[INFO] Identifier Parent: coordinator
[INFO] IP: 10.100.1.243
[INFO] Port: 5432
[INFO] FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 90feffd7-89fb-4afb-a63f-cc975d7e928c
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[1;32m[OK][0m ✓ PostgreSQL coordinator registered
[INFO] Registering PgBouncer to monitoring dashboard...
[INFO] FQDN: db-web-universe-main-dev-postgresql-bouncer.fastorder.com, IP: 10.100.1.244, Port: 6432
[INFO] Key permissions set for www-data access
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: PgBouncer
[INFO] Identifier: web-universe-main-dev-pgbouncer
[INFO] Identifier Parent: pooling
[INFO] IP: 10.100.1.244
[INFO] Port: 6432
[INFO] FQDN: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: fdc70c5f-615d-432f-8161-a7acd56ea9ed
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[1;32m[OK][0m ✓ PgBouncer registered
[INFO] 5️⃣ Creating PgBouncer professional monitoring rules...
[INFO] Creating PgBouncer recording rules...
[1;32m[OK][0m ✓ PgBouncer recording rules created
[INFO] Creating PgBouncer alert rules with runbook URLs...
[1;32m[OK][0m ✓ PgBouncer alert rules with runbook URLs created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️ Could not reload Prometheus (may need manual reload)
[INFO] 6️⃣ Setting up pgbackrest_exporter integration...
[INFO] pgBackRest detected, setting up exporter...
[1;32m[OK][0m ✓ pgbackrest_exporter already installed
[INFO] Creating pgbackrest_exporter systemd service...
[1;32m[OK][0m ✓ pgbackrest_exporter service file created
[INFO] Starting pgbackrest_exporter service...
[2026-02-05 07:22:41 UTC] USER=www-data EUID=0 PID=178794 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 07:22:42 UTC] USER=www-data EUID=0 PID=178867 ACTION=passthru ARGS=systemctl enable pgbackrest_exporter-web-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/pgbackrest_exporter-web-universe-main-dev.service → /etc/systemd/system/pgbackrest_exporter-web-universe-main-dev.service.
[2026-02-05 07:22:42 UTC] USER=www-data EUID=0 PID=178928 ACTION=passthru ARGS=systemctl restart pgbackrest_exporter-web-universe-main-dev.service
[WARN] ⚠️ pgbackrest_exporter service not running (may need manual start)
[WARN] Run: systemctl status pgbackrest_exporter-web-universe-main-dev.service
[INFO] Creating pgBackRest alert rules...
[1;32m[OK][0m ✓ pgBackRest alert rules created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️ Could not reload Prometheus (may need manual reload)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ PostgreSQL & PgBouncer Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] postgres_exporter: http://localhost:9187/metrics
[INFO] pgbouncer_exporter: http://localhost:9127/metrics
[INFO] Prometheus: https://metrics-web-universe-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-web-universe-main-dev.fastorder.com
[INFO]
[INFO] PgBouncer Monitoring:
[INFO] • Recording rules: /etc/prometheus/obs-web-universe-main-dev/rules/pgbouncer_recording_rules.yml
[INFO] • Alert rules: /etc/prometheus/obs-web-universe-main-dev/rules/pgbouncer_alerts.yml
[INFO]
[INFO] pgBackRest Monitoring:
[INFO] • Alert rules: /etc/prometheus/obs-web-universe-main-dev/rules/pgbackrest_alerts.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up monitoring for 1 worker(s) and 0 standby(s) per worker...
[0;34m[INFO][0m Setting up monitoring for: worker-01
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;32m✓ [SECRETS][0m Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[0;34m[SECRETS][0m Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[0;34m[SECRETS][0m Search (build_es_secret_name, get/set_es_credentials_to_vault)
[0;34m[SECRETS][0m Backups (build_backup_path)
[0;34m[SECRETS][0m Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 PostgreSQL Monitoring Integration for web-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-web-universe-main-dev
[1;32m[OK][0m Observability cell endpoints registered for web-universe-main-dev
[1;32m[OK][0m ✓ Observability cell is ready
[INFO] ✓ Using private IP for metrics: 10.100.1.243
[INFO] 2️⃣ Setting up postgres_exporter integration...
[INFO] Checking observability cell readiness: obs-web-universe-main-dev
[1;32m[OK][0m Observability cell endpoints registered for web-universe-main-dev
[INFO] Setting up postgres_exporter for web-universe-main-dev
[2026-02-05 07:22:46 UTC] USER=www-data EUID=0 PID=179208 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-web-universe-main-dev.yaml /etc/prometheus/postgres_exporter_queries-web-universe-main-dev.yaml
[2026-02-05 07:22:46 UTC] USER=www-data EUID=0 PID=179218 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-web-universe-main-dev.yaml
[2026-02-05 07:22:46 UTC] USER=www-data EUID=0 PID=179230 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-web-universe-main-dev.yaml
[1;32m[OK][0m Custom queries file created at /etc/prometheus/postgres_exporter_queries-web-universe-main-dev.yaml
[1;32m[OK][0m postgres_exporter already running with custom queries for web-universe-main-dev
[1;32m[OK][0m ✓ postgres_exporter integration complete
[INFO] Registering postgres_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: postgres_exporter -> 10.100.1.243:9187
[1;32m[OK][0m ✓ Registered postgres_exporter scrape target: 10.100.1.243:9187
[INFO] Target file: /etc/prometheus/obs-web-universe-main-dev/targets/postgres_exporter.yml
[1;32m[OK][0m ✓ postgres_exporter registered as Prometheus scrape target
[INFO] 3️⃣ Setting up pgbouncer_exporter integration...
[INFO] PgBouncer FQDN found in /etc/hosts: db-web-universe-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.244
[INFO] PgBouncer detected: db-web-universe-main-dev-postgresql-bouncer.fastorder.com:6432
[1;32m[OK][0m ✓ pgbouncer_exporter already installed
[INFO] Getting pgbouncer_admin password (SERVICE=web, ZONE=universe)
[1;32m[OK][0m ✓ pgbouncer_admin password retrieved (24 chars)
[INFO] Using pgbouncer certs from: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[INFO] Creating pgbouncer_exporter systemd service...
[1;32m[OK][0m ✓ pgbouncer_exporter service file created
[INFO] Starting pgbouncer_exporter service...
[2026-02-05 07:22:48 UTC] USER=www-data EUID=0 PID=179326 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 07:22:48 UTC] USER=www-data EUID=0 PID=179375 ACTION=passthru ARGS=systemctl enable pgbouncer_exporter-web-universe-main-dev.service
[2026-02-05 07:22:49 UTC] USER=www-data EUID=0 PID=179440 ACTION=passthru ARGS=systemctl restart pgbouncer_exporter-web-universe-main-dev.service
[1;32m[OK][0m ✓ pgbouncer_exporter service running
[INFO] Registering pgbouncer_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: pgbouncer_exporter -> 10.100.1.244:9127
[1;32m[OK][0m ✓ Registered pgbouncer_exporter scrape target: 10.100.1.244:9127
[INFO] Target file: /etc/prometheus/obs-web-universe-main-dev/targets/pgbouncer_exporter.yml
[1;32m[OK][0m ✓ pgbouncer_exporter registered as Prometheus scrape target
[INFO] 4️⃣ Registering nodes to monitoring database...
[INFO] PostgreSQL key permissions set for www-data access: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[INFO] Registering PostgreSQL worker-01 to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: PostgreSQL
[INFO] Identifier: web-universe-main-dev-postgresql-worker-01
[INFO] Identifier Parent: worker-01
[INFO] IP: 10.100.1.243
[INFO] Port: 5432
[INFO] FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 84e4185a-2ef1-49c1-8d2a-841d077f036b
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[1;32m[OK][0m ✓ PostgreSQL worker-01 registered
[INFO] Registering PgBouncer to monitoring dashboard...
[INFO] FQDN: db-web-universe-main-dev-postgresql-bouncer.fastorder.com, IP: 10.100.1.244, Port: 6432
[INFO] Key permissions set for www-data access
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: PgBouncer
[INFO] Identifier: web-universe-main-dev-pgbouncer
[INFO] Identifier Parent: pooling
[INFO] IP: 10.100.1.244
[INFO] Port: 6432
[INFO] FQDN: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: fdc70c5f-615d-432f-8161-a7acd56ea9ed
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[1;32m[OK][0m ✓ PgBouncer registered
[INFO] 5️⃣ Creating PgBouncer professional monitoring rules...
[INFO] Creating PgBouncer recording rules...
[1;32m[OK][0m ✓ PgBouncer recording rules created
[INFO] Creating PgBouncer alert rules with runbook URLs...
[1;32m[OK][0m ✓ PgBouncer alert rules with runbook URLs created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️ Could not reload Prometheus (may need manual reload)
[INFO] 6️⃣ Setting up pgbackrest_exporter integration...
[INFO] pgBackRest detected, setting up exporter...
[1;32m[OK][0m ✓ pgbackrest_exporter already installed
[INFO] Creating pgbackrest_exporter systemd service...
[1;32m[OK][0m ✓ pgbackrest_exporter service file created
[INFO] Starting pgbackrest_exporter service...
[2026-02-05 07:22:54 UTC] USER=www-data EUID=0 PID=179714 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 07:22:54 UTC] USER=www-data EUID=0 PID=179760 ACTION=passthru ARGS=systemctl enable pgbackrest_exporter-web-universe-main-dev.service
[2026-02-05 07:22:55 UTC] USER=www-data EUID=0 PID=179810 ACTION=passthru ARGS=systemctl restart pgbackrest_exporter-web-universe-main-dev.service
[WARN] ⚠️ pgbackrest_exporter service not running (may need manual start)
[WARN] Run: systemctl status pgbackrest_exporter-web-universe-main-dev.service
[INFO] Creating pgBackRest alert rules...
[1;32m[OK][0m ✓ pgBackRest alert rules created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️ Could not reload Prometheus (may need manual reload)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ PostgreSQL & PgBouncer Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] postgres_exporter: http://localhost:9187/metrics
[INFO] pgbouncer_exporter: http://localhost:9127/metrics
[INFO] Prometheus: https://metrics-web-universe-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-web-universe-main-dev.fastorder.com
[INFO]
[INFO] PgBouncer Monitoring:
[INFO] • Recording rules: /etc/prometheus/obs-web-universe-main-dev/rules/pgbouncer_recording_rules.yml
[INFO] • Alert rules: /etc/prometheus/obs-web-universe-main-dev/rules/pgbouncer_alerts.yml
[INFO]
[INFO] pgBackRest Monitoring:
[INFO] • Alert rules: /etc/prometheus/obs-web-universe-main-dev/rules/pgbackrest_alerts.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✓[0m ✅ Monitoring setup completed for coordinator, workers, and standbys
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 12-setup-offsite-backup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Setting up offsite backup repository for web-universe-main-dev...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Offsite Backup Repository Setup (repo2)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📋 OFFSITE BACKUP INFORMATION
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Why Offsite Backups?
[INFO] ✓ Disaster recovery resilience (datacenter loss, hardware failure)
[INFO] ✓ Protection against local corruption or ransomware
[INFO] ✓ Compliance requirements (geographic redundancy)
[INFO] ✓ Long-term archival with cost-effective storage tiers
[WARN] ⚠️ Offsite backup (repo2) is NOT ENABLED
[WARN] Using local backups only (repo1)
[INFO] Configuration Example Location:
[INFO] 📄 /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[INFO] Supported Storage Backends:
[INFO] • AWS S3 (standard, multi-region)
[INFO] • AWS S3 Glacier (low-cost archival)
[INFO] • MinIO (self-hosted S3-compatible)
[INFO] • Google Cloud Storage (via S3 compatibility)
[INFO] • Azure Blob Storage (via S3 compatibility)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📝 SETUP INSTRUCTIONS
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Step 1: Review the example configuration
[INFO] cat /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[INFO] Step 2: Prepare S3 bucket and credentials
[INFO] • Create S3 bucket (or MinIO bucket)
[INFO] • Create IAM user with S3 permissions (PutObject, GetObject, DeleteObject, ListBucket)
[INFO] • Note: Access Key ID and Secret Access Key
[INFO] Step 3: Add repo2 configuration to /etc/pgbackrest/pgbackrest.conf
[INFO] • Copy repo2-* settings from example to [global] section
[INFO] • Replace placeholders (bucket name, access keys, region)
[INFO] • Note: Use same cipher key as repo1, or generate separate key for repo2
[INFO] Step 4: Initialize repo2 stanzas
[INFO] command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator stanza-create --repo=2
[INFO] command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=web-universe-main-dev-worker-01 stanza-create --repo=2
[INFO] command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=web-universe-main-dev-worker-02 stanza-create --repo=2
[INFO] Step 5: Verify repo2 configuration
[INFO] command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator check --repo=2
[INFO] Step 6: Take initial full backup to repo2
[INFO] command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator --repo=2 --type=full backup
[INFO] Step 7: Update backup automation to include repo2
[INFO] • Edit: /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[INFO] • Change: pgbackrest backup to pgbackrest --repo=1,2 backup
[INFO] • Or: Add separate cron for repo2 backups
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🧪 TESTING
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] After configuration, run:
[INFO] ./08-setup-offsite-backup.sh test
[INFO] This will verify:
[INFO] ✓ S3 connectivity
[INFO] ✓ Stanza initialization
[INFO] ✓ Test backup and restore from repo2
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 💡 COST OPTIMIZATION
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] AWS S3 Lifecycle Policies (transition to cheaper storage):
[INFO] • 0-30 days: S3 Standard (~$0.023/GB/month)
[INFO] • 30-90 days: S3 Standard-IA (~$0.0125/GB/month)
[INFO] • 90+ days: S3 Glacier (~$0.004/GB/month)
[INFO] Estimated costs for 100GB backups:
[INFO] • All Standard: ~$2.30/month
[INFO] • With lifecycle: ~$1.20/month (48% savings)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 13-setup-monitoring-alerts.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Setting up backup monitoring and alerting for web-universe-main-dev...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Monitoring and Alerting Configuration
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] This will set up monitoring for:
• Backup failures (cron job failures)
• WAL archiving backlog (>100 files)
• Repository disk space (<20% free)
• Backup age (>25 hours)
[INFO] No alert email configured (set ALERT_EMAIL environment variable)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1️⃣ Creating monitoring directories...
[2026-02-05 07:23:02 UTC] USER=www-data EUID=0 PID=180125 ACTION=fsop ARGS=mkdir -p /opt/pgbackrest-monitoring
[2026-02-05 07:23:02 UTC] USER=www-data EUID=0 PID=180134 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest-monitoring
[2026-02-05 07:23:02 UTC] USER=www-data EUID=0 PID=180143 ACTION=fsop ARGS=chmod 777 /opt/pgbackrest-monitoring
[2026-02-05 07:23:02 UTC] USER=www-data EUID=0 PID=180155 ACTION=fsop ARGS=chmod 777 /var/log/pgbackrest-monitoring
[2026-02-05 07:23:03 UTC] USER=www-data EUID=0 PID=180166 ACTION=fsop ARGS=chown postgres:postgres /opt/pgbackrest-monitoring
[INFO] ✅ Directories created
[INFO] 2️⃣ Creating alert helper script...
[2026-02-05 07:23:03 UTC] USER=www-data EUID=0 PID=180219 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/send-alert.sh
[INFO] ✅ Alert helper created
[INFO] 3️⃣ Creating WAL queue monitoring script...
[2026-02-05 07:23:03 UTC] USER=www-data EUID=0 PID=180256 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-wal-queue.sh
[INFO] ✅ WAL queue monitor created
[INFO] 4️⃣ Creating backup age monitoring script...
[INFO] ✅ Backup age monitor created
[INFO] 5️⃣ Creating repository disk space monitoring script...
[2026-02-05 07:23:04 UTC] USER=www-data EUID=0 PID=180302 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-repo-space.sh
[INFO] ✅ Disk space monitor created
[INFO] 6️⃣ Creating backup failure detection script...
[2026-02-05 07:23:04 UTC] USER=www-data EUID=0 PID=180323 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-backup-failures.sh
[INFO] ✅ Backup failure detector created
[INFO] 7️⃣ Creating master monitoring script...
[2026-02-05 07:23:04 UTC] USER=www-data EUID=0 PID=180341 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO] ✅ Master monitoring script created
[INFO] 8️⃣ Installing mailutils for email alerts...
[INFO] ✅ mailutils already installed
[INFO] 9️⃣ Installing jq for JSON parsing...
[INFO] ✅ jq already installed
[INFO] 🔟 Setting up monitoring cron jobs...
[2026-02-05 07:23:04 UTC] USER=www-data EUID=0 PID=180359 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-monitoring-web-universe-main-dev
[INFO] ✅ Monitoring cron jobs configured
[INFO] Checks run every 15 minutes
[INFO] 1️⃣1️⃣ Creating monitoring dashboard...
[2026-02-05 07:23:04 UTC] USER=www-data EUID=0 PID=180379 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/dashboard.sh
[INFO] ✅ Monitoring dashboard created
[INFO] 1️⃣2️⃣ Running initial monitoring check...
[2026-02-05 07:23:04 UTC] USER=www-data EUID=0 PID=180388 ACTION=passthru ARGS=bash /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup monitoring setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Monitoring Configuration:
[INFO] Alert Email:
[INFO] Slack Webhook: Not configured
[INFO] Monitoring Checks:
[INFO] • WAL Queue: Every 15 minutes (threshold: >100 files)
[INFO] • Backup Age: Every 15 minutes (threshold: >25 hours)
[INFO] • Disk Space: Every 15 minutes (threshold: <20% free)
[INFO] • Backup Failures: Every 15 minutes (log analysis)
[INFO] Scripts Created:
[INFO] Monitoring dir: /opt/pgbackrest-monitoring
[INFO] Log dir: /var/log/pgbackrest-monitoring
[INFO] Dashboard: /opt/pgbackrest-monitoring/dashboard.sh
[INFO] Master check: /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO] Alert sender: /opt/pgbackrest-monitoring/send-alert.sh
[INFO] Useful Commands:
[INFO] View dashboard: /usr/local/bin/fastorder-provisioning-wrapper.sh /opt/pgbackrest-monitoring/dashboard.sh
[INFO] Run checks now: /usr/local/bin/fastorder-provisioning-wrapper.sh /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO] View alerts: tail -f /var/log/pgbackrest-monitoring/alerts.log
[INFO] View monitoring: tail -f /var/log/pgbackrest-monitoring/monitoring.log
[INFO] Cron Schedule:
[INFO] All checks: Every 15 minutes
[INFO] Log rotation: Weekly (keep 7 days)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 14-vault-cipher-key.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ✅ Using permanent AWS credentials from /home/ab/.aws/credentials [default] profile
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔐 PostgreSQL Cipher Key Vaulting
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Environment: web-universe-main-dev
[INFO] AWS Region: me-central-1
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1️⃣ Verifying AWS setup...
[INFO] ✅ AWS authentication successful
[INFO] 2️⃣ Verifying cipher key...
[INFO] ✅ Cipher key found
[INFO] Location: /etc/pgbackrest/.cipher-key-web-universe-main-dev
[INFO] Hash (MD5): be79d8ccabfdb669f39e8337f0dd317a
[INFO] Size: 191 bytes
[INFO] 3️⃣ Vaulting cipher key to AWS Secrets Manager...
[INFO] Secret name: fastorder/db/web/universe/main/dev/postgresql/pgbackrest/cipher-key
[INFO] Creating new secret...
[INFO] ✅ Cipher key stored in AWS Secrets Manager
[INFO] Verifying storage...
[INFO] ✅ Verification successful - key matches
[INFO] 4️⃣ Creating local encrypted backup...
[2026-02-05 07:23:14 UTC] USER=www-data EUID=0 PID=180823 ACTION=fsop ARGS=mv /tmp/cipher-key-backup-180593.enc /root/.pgbackrest-cipher-key-web-universe-main-dev.enc
[2026-02-05 07:23:14 UTC] USER=www-data EUID=0 PID=180834 ACTION=fsop ARGS=chmod 600 /root/.pgbackrest-cipher-key-web-universe-main-dev.enc
[2026-02-05 07:23:15 UTC] USER=www-data EUID=0 PID=180853 ACTION=fsop ARGS=chmod 600 /root/.pgbackrest-cipher-key-passphrase-web-universe-main-dev.txt
[INFO] ✅ Local encrypted backup created
[INFO] Backup file: /root/.pgbackrest-cipher-key-web-universe-main-dev.enc
[INFO] Passphrase: /root/.pgbackrest-cipher-key-passphrase-web-universe-main-dev.txt
[INFO] 5️⃣ Vaulting backup passphrase...
[INFO] ✅ Backup passphrase vaulted
[INFO] 6️⃣ Creating recovery documentation...
[2026-02-05 07:23:18 UTC] USER=www-data EUID=0 PID=181017 ACTION=fsop ARGS=chmod 640 /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_web-universe-main-dev.md
[2026-02-05 07:23:18 UTC] USER=www-data EUID=0 PID=181027 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_web-universe-main-dev.md
[INFO] ✅ Recovery documentation: /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_web-universe-main-dev.md
[INFO] 7️⃣ Storing backup metadata...
[INFO] ✅ Backup metadata stored in AWS Secrets Manager
[INFO] Secret: fastorder/db/web/universe/main/dev/postgresql/backup/metadata-20260205-072318
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Cipher Key Vaulting Complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Environment: web-universe-main-dev
[INFO] Key Hash: be79d8ccabfdb669f39e8337f0dd317a
[INFO] AWS Secrets:
[INFO] Cipher Key: fastorder/db/web/universe/main/dev/postgresql/pgbackrest/cipher-key
[INFO] Passphrase: fastorder/db/web/universe/main/dev/postgresql/pgbackrest/cipher-key-passphrase
[INFO] Backup Metadata: fastorder/db/web/universe/main/dev/postgresql/backup/metadata-20260205-072318
[INFO] Local Backups:
[INFO] Encrypted File: /root/.pgbackrest-cipher-key-web-universe-main-dev.enc
[INFO] Passphrase File: /root/.pgbackrest-cipher-key-passphrase-web-universe-main-dev.txt
[INFO] Recovery Doc: /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_web-universe-main-dev.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Skipping 15-backup-restore-test.sh (test script - set RUN_TESTS=true to enable)
[0;34m[INFO][0m Skipping 16-test-recovery.sh (test script - set RUN_TESTS=true to enable)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 17-verification.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] PostgreSQL Production Readiness Verification
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO]
[INFO] This script verifies 3 CRITICAL checks for production readiness:
[INFO] 1. Citus Cluster Operational (coordinator + workers)
[INFO] 2. SSL/TLS Enforced (certificates valid, connections secure)
[INFO] 3. Coordinator Backups Configured (pgBackRest functional)
[INFO]
[INFO] 📖 Documentation: /tmp/VERIFICATION_RUNBOOK.md
[INFO] 🔐 Security: Uses sudo for certificate checks (maintains strict permissions)
[INFO] 📊 Exit Code: 0 = production ready, 1 = critical checks failed
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] 🕐 Ensuring all PostgreSQL services are ready...
[ OK ] ✅ All PostgreSQL services are ready
[INFO] 🔍 Starting PostgreSQL verification...
[INFO] Environment: web-universe-main-dev
[INFO] Citus: yes
[INFO] Citus mode ENABLED
[INFO] → Coordinator + 1 worker(s) + 3 HA node(s) per worker
[INFO] Verifying 1 worker(s)...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying: worker-01 (type: worker-01)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m 🔍 Starting PostgreSQL verification for web-universe-main-dev-worker-01...
[0;34m[INFO][0m 1️⃣ Checking systemd service status...
[0;32m[OK][0m ✅ Service postgresql@web-universe-main-dev-worker-01.service is active
[0;34m[INFO][0m 2️⃣ Checking PostgreSQL process...
[0;32m[OK][0m ✅ PostgreSQL process is running
[0;34m[INFO][0m 3️⃣ Checking socket directory...
[0;32m[OK][0m ✅ Socket directory exists: /var/run/postgresql-web-universe-main-dev-worker-01
total 4
drwxrwsr-x 2 postgres postgres 80 Feb 5 07:19 .
drwxr-xr-x 61 root root 1620 Feb 5 07:23 ..
srwxrwxrwx 1 postgres postgres 0 Feb 5 07:19 .s.PGSQL.5432
-rw------- 1 postgres postgres 127 Feb 5 07:19 .s.PGSQL.5432.lock
[0;34m[INFO][0m 4️⃣ Testing connection via Unix socket...
[0;32m[OK][0m ✅ Socket connection successful
version
-----------------------------------------------------------------------------------------------------------------------------------
PostgreSQL 17.6 (Ubuntu 17.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
[0;34m[INFO][0m 5️⃣ Checking SSL certificates...
[2026-02-05 07:23:22 UTC] USER=www-data EUID=0 PID=181247 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[0;32m[OK][0m ✅ Server certificate exists: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 07:23:22 UTC] USER=www-data EUID=0 PID=181256 ACTION=fsop ARGS=openssl x509 -in /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt -noout -checkend 86400
Certificate will not expire
[0;32m[OK][0m ✅ Server certificate is valid
[2026-02-05 07:23:22 UTC] USER=www-data EUID=0 PID=181265 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[0;32m[OK][0m ✅ CA certificate exists: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[0;34m[INFO][0m ℹ️ Client certificates not found at /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[0;34m[INFO][0m (This is OK if using password authentication)
[0;34m[INFO][0m 6️⃣ Checking PostgreSQL settings...
[0;32m[OK][0m ✅ SSL is enabled worker-01 worker-01
[0;32m[OK][0m ✅ Max connections: 100
[0;32m[OK][0m ✅ Listen addresses: 10.100.1.243
[0;32m[OK][0m ✅ WAL level: logical
[0;32m[OK][0m ✅ Shared preload libraries: shared_preload_libraries
[0;34m[INFO][0m 7️⃣ Checking replication configuration...
[0;34m[INFO][0m ℹ️ No synchronous standbys configured (single node or async replication)
[0;34m[INFO][0m Checking replication slots...
slot_name | slot_type | active | restart_lsn
-----------+-----------+--------+-------------
(0 rows)
[0;32m[OK][0m ✅ Replication slot naming uses underscores (correct)
[0;34m[INFO][0m Checking active replication connections...
application_name | client_addr | state | sync_state
------------------+-------------+-------+------------
(0 rows)
[0;34m[INFO][0m ℹ️ No active replication connections
[0;34m[INFO][0m ℹ️ This is a PRIMARY node (no standby.signal)
[0;34m[INFO][0m 8️⃣ Checking pg_hba.conf for replication rules...
[1;33m[WARN][0m ⚠️ pg_hba.conf not found at /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
[0;34m[INFO][0m 9️⃣ Checking Citus configuration...
[0;32m[OK][0m ✅ Citus extension is installed
[0;32m[OK][0m ✅ Citus version: Citus 13.2.0
[0;32m[OK][0m ✅ max_prepared_transactions: 100 (adequate for Citus)
[0;34m[INFO][0m Citus active worker nodes:
node_name | node_port
-------------------------------------------------------------+-----------
db-web-universe-main-dev-postgresql-worker-01.fastorder.com | 5432
(1 row)
[0;34m[INFO][0m 🔟 Checking data directory...
[0;32m[OK][0m ✅ Data directory exists: /data/postgresql/17/web-universe-main-dev/worker-01
[0;32m[OK][0m ✅ Data directory size: 4.0K
[0;34m[INFO][0m 1️⃣1️⃣ Checking PgBouncer configuration...
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini' as root on web-03.
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/%i/pgbouncer.ini' as root on web-03.
[0;32m[OK][0m ✅ PgBouncer is installed
[0;34m[INFO][0m Version: 1.24.1
2.1.12-stable
c-ares
OpenSSL
yes
[0;32m[OK][0m ✅ PgBouncer service is active: pgbouncer@web-universe-main-dev.service
[1;33m[WARN][0m ⚠️ PgBouncer IP service is not active: pgbouncer-ip@web-universe-main-dev.service
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini' as root on web-03.
[1;33m[WARN][0m ⚠️ PgBouncer config not found: /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/web-universe-main-dev/userlist.txt' as root on web-03.
[1;33m[WARN][0m ⚠️ PgBouncer auth file not found: /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;32m[OK][0m ✅ PgBouncer is listening on port 6432
[0;34m[INFO][0m 1️⃣2️⃣ Enhanced PgBouncer Admin Console Verification...
Failed to print table: Broken pipe
[0;34m[INFO][0m ℹ️ PgBouncer not configured for enhanced verification
[0;34m[INFO][0m 1️⃣3️⃣ Replicator User Connection Verification...
[0;34m[INFO][0m ℹ️ No replication slots configured - skipping replicator verification
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ PostgreSQL verification completed successfully!
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Instance: web-universe-main-dev-worker-01
[0;34m[INFO][0m Service: postgresql@web-universe-main-dev-worker-01.service
[0;34m[INFO][0m Socket: /var/run/postgresql-web-universe-main-dev-worker-01
[0;34m[INFO][0m Data Directory: /data/postgresql/17/web-universe-main-dev/worker-01
[0;34m[INFO][0m Hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[0;34m[INFO][0m Port: 5432
[0;34m[INFO][0m SSL: on
[0;34m[INFO][0m WAL Level: logical
[0;34m[INFO][0m Citus: yes
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Failed to print table: Broken pipe
[ OK ] ✅ Verification passed for worker-01
[INFO] Skipping worker-01-standby-01 - service not configured
[INFO] Skipping worker-01-standby-02 - service not configured
[INFO] Skipping worker-01-standby-03 - service not configured
[INFO] Verifying coordinator...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying: coordinator (type: coordinator)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m 🔍 Starting PostgreSQL verification for web-universe-main-dev-coordinator...
[0;34m[INFO][0m 1️⃣ Checking systemd service status...
[0;32m[OK][0m ✅ Service postgresql@web-universe-main-dev-coordinator.service is active
[0;34m[INFO][0m 2️⃣ Checking PostgreSQL process...
[0;32m[OK][0m ✅ PostgreSQL process is running
[0;34m[INFO][0m 3️⃣ Checking socket directory...
[0;32m[OK][0m ✅ Socket directory exists: /var/run/postgresql-web-universe-main-dev-coordinator
total 4
drwxrwsr-x 2 postgres postgres 80 Feb 5 07:21 .
drwxr-xr-x 65 root root 1700 Feb 5 07:23 ..
srwxrwxrwx 1 postgres postgres 0 Feb 5 07:21 .s.PGSQL.5432
-rw------- 1 postgres postgres 131 Feb 5 07:21 .s.PGSQL.5432.lock
[0;34m[INFO][0m 4️⃣ Testing connection via Unix socket...
[0;32m[OK][0m ✅ Socket connection successful
version
-----------------------------------------------------------------------------------------------------------------------------------
PostgreSQL 17.6 (Ubuntu 17.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
[0;34m[INFO][0m 5️⃣ Checking SSL certificates...
[0;32m[OK][0m ✅ Server certificate exists: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 07:23:39 UTC] USER=www-data EUID=0 PID=182183 ACTION=fsop ARGS=openssl x509 -in /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt -noout -checkend 86400
Certificate will not expire
[0;32m[OK][0m ✅ Server certificate is valid
[2026-02-05 07:23:39 UTC] USER=www-data EUID=0 PID=182192 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[0;32m[OK][0m ✅ CA certificate exists: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[0;34m[INFO][0m ℹ️ Client certificates not found at /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[0;34m[INFO][0m (This is OK if using password authentication)
[0;34m[INFO][0m 6️⃣ Checking PostgreSQL settings...
[0;32m[OK][0m ✅ SSL is enabled coordinator coordinator
[0;32m[OK][0m ✅ Max connections: 150
[0;32m[OK][0m ✅ Listen addresses: 10.100.1.54
[0;32m[OK][0m ✅ WAL level: logical
[0;32m[OK][0m ✅ Shared preload libraries: shared_preload_libraries
[0;34m[INFO][0m 7️⃣ Checking replication configuration...
[0;34m[INFO][0m ℹ️ No synchronous standbys configured (single node or async replication)
[0;34m[INFO][0m Checking replication slots...
slot_name | slot_type | active | restart_lsn
----------------------------------+-----------+--------+-------------
dbz_web_universe_main_dev_config | logical | f | 0/700E1D8
(1 row)
[0;32m[OK][0m ✅ Replication slot naming uses underscores (correct)
[0;34m[INFO][0m Checking active replication connections...
application_name | client_addr | state | sync_state
------------------+-------------+-------+------------
(0 rows)
[0;34m[INFO][0m ℹ️ No active replication connections
[0;34m[INFO][0m ℹ️ This is a PRIMARY node (no standby.signal)
[0;34m[INFO][0m 8️⃣ Checking pg_hba.conf for replication rules...
[1;33m[WARN][0m ⚠️ pg_hba.conf not found at /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
[0;34m[INFO][0m 9️⃣ Checking Citus configuration...
[0;32m[OK][0m ✅ Citus extension is installed
[0;32m[OK][0m ✅ Citus version: Citus 13.2.0
[0;32m[OK][0m ✅ max_prepared_transactions: 100 (adequate for Citus)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m COORDINATOR-SPECIFIC CHECKS
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Checking registered workers...
[0;34m[INFO][0m ℹ️ Coordinator role verified via pg_dist_node (1 workers registered)
[0;34m[INFO][0m Checking coordinator hostname configuration...
[0;32m[OK][0m ✅ Coordinator hostname: ---------------------------------------------------------------:----------
[0;34m[INFO][0m Checking for stuck prepared transactions...
[0;32m[OK][0m ✅ No stuck Citus prepared transactions
[0;34m[INFO][0m Expected workers: 1
[0;34m[INFO][0m Registered workers: 1
[0;32m[OK][0m ✅ All 1 worker(s) successfully registered
[0;34m[INFO][0m Registered worker nodes:
nodename | nodeport | groupid | isactive | noderole | shouldhaveshards
---------------------------------------------------------------+----------+---------+----------+----------+------------------
db-web-universe-main-dev-postgresql-coordinator.fastorder.com | 5432 | 0 | t | primary | f
db-web-universe-main-dev-postgresql-worker-01.fastorder.com | 5432 | 1 | t | primary | t
(2 rows)
[0;34m[INFO][0m Note: groupid=0 is the coordinator, groupid>0 are workers
[0;34m[INFO][0m Citus active worker nodes:
psql: error: connection to server at "db-web-universe-main-dev-postgresql.fastorder.com" (10.100.1.54), port 5432 failed: SSL error: certificate verify failed
connection to server at "db-web-universe-main-dev-postgresql.fastorder.com" (10.100.1.54), port 5432 failed: FATAL: pg_hba.conf rejects connection for host "10.100.1.54", user "postgres", database "postgres", no encryption
[0;34m[INFO][0m Verifying Citus workers...
[0;34m[INFO][0m Checking worker: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
citus_add_node
----------------
2
(1 row)
[0;34m[INFO][0m Testing Citus distributed table setup...
[0;34m[INFO][0m Checking for blocking locks...
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE pid <> pg_backend_pid()
AND state = 'idle in transaction'
AND query_start < NOW() - INTERVAL '30 seconds'
AND datname = current_database();
pg_terminate_backend
----------------------
(0 rows)
[0;34m[INFO][0m Creating demo schema (if needed)...
CREATE SCHEMA
[0;32m[OK][0m ✅ Demo schema ready
[0;34m[INFO][0m Creating distributed table 'demo.events'...
CREATE TABLE
[0;32m[OK][0m ✅ Table is already distributed
[0;34m[INFO][0m Inserting test data...
INSERT 0 1
[0;32m[OK][0m ✅ Distributed table contains 1 row(s)
[0;34m[INFO][0m Checking shard distribution...
[0;32m[OK][0m ✅ Table has 1 shard(s)
[0;34m[INFO][0m Shard placement across workers (first 10 shards):
shardid | nodename | nodeport | shardstate
---------+----------+----------+------------
(0 rows)
[0;32m[OK][0m ✅ Verified 3 shard placement(s)
[0;34m[INFO][0m Testing query routing (EXPLAIN for user_id=42)...
[0;34m[INFO][0m Query plan: QUERY PLAN
--------------------------
Seq Scan on events
Filter: (user_id = 42)
(2 rows)
[0;34m[INFO][0m 🔟 Checking data directory...
[0;32m[OK][0m ✅ Data directory exists: /data/postgresql/17/web-universe-main-dev/coordinator
[0;32m[OK][0m ✅ Data directory size: 4.0K
[0;34m[INFO][0m 1️⃣1️⃣ Checking PgBouncer configuration...
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini' as root on web-03.
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/%i/pgbouncer.ini' as root on web-03.
[0;32m[OK][0m ✅ PgBouncer is installed
[0;34m[INFO][0m Version: 1.24.1
2.1.12-stable
c-ares
OpenSSL
yes
[0;32m[OK][0m ✅ PgBouncer service is active: pgbouncer@web-universe-main-dev.service
[1;33m[WARN][0m ⚠️ PgBouncer IP service is not active: pgbouncer-ip@web-universe-main-dev.service
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini' as root on web-03.
[1;33m[WARN][0m ⚠️ PgBouncer config not found: /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/web-universe-main-dev/userlist.txt' as root on web-03.
[1;33m[WARN][0m ⚠️ PgBouncer auth file not found: /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;32m[OK][0m ✅ PgBouncer is listening on port 6432
[0;34m[INFO][0m 1️⃣2️⃣ Enhanced PgBouncer Admin Console Verification...
[0;34m[INFO][0m ℹ️ PgBouncer password not found
[0;34m[INFO][0m 1️⃣3️⃣ Replicator User Connection Verification...
[0;34m[INFO][0m Found 1 replication slot(s) - verifying replicator connectivity...
[1;33m[WARN][0m ⚠️ Replicator certificates not found at /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[0;34m[INFO][0m Expected files:
[0;34m[INFO][0m - /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[0;34m[INFO][0m - /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/replicator.crt
[0;34m[INFO][0m - /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/replicator.key
[0;34m[INFO][0m Checking pg_hba.conf replicator rules...
[0;32m[OK][0m ✅ Replicator HBA rules found:
line_number | type | database | user_name | address | auth_method | options | error
-------------+------+----------+-----------+---------+-------------+---------+-------
(0 rows)
[0;34m[INFO][0m Checking active replicator connections in pg_stat_activity...
[1;33m[WARN][0m ⚠️ No active replicator connections in pg_stat_activity
[1;33m[WARN][0m This is expected if standbys are not currently connected
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ PostgreSQL verification completed successfully!
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Instance: web-universe-main-dev-coordinator
[0;34m[INFO][0m Service: postgresql@web-universe-main-dev-coordinator.service
[0;34m[INFO][0m Socket: /var/run/postgresql-web-universe-main-dev-coordinator
[0;34m[INFO][0m Data Directory: /data/postgresql/17/web-universe-main-dev/coordinator
[0;34m[INFO][0m Hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[0;34m[INFO][0m Port: 5432
[0;34m[INFO][0m SSL: on
[0;34m[INFO][0m WAL Level: logical
[0;34m[INFO][0m Citus: yes
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Failed to print table: Broken pipe
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 💡 OPTIMIZATION OPPORTUNITIES (Optional Enhancements)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 1. Enable PgBouncer connection pooling
[0;34m[INFO][0m 📦 Benefit: Reduces connection overhead for high-concurrency workloads
[0;34m[INFO][0m ⚡ Use case: When facing connection exhaustion or frequent connect/disconnect cycles
[0;34m[INFO][0m 🔧 Action: Enable and configure pgbouncer@web-universe-main-dev.service
[0;34m[INFO][0m 📚 Docs: https://www.pgbouncer.org/config.html
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ℹ️ These are optional enhancements for production-scale deployments
[0;34m[INFO][0m ℹ️ Current configuration is fully functional and ready for production
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] ✅ Verification passed for coordinator
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📊 PRODUCTION READINESS CHECKS (Step 04 & 05)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Checking Monitoring Setup (postgres_exporter or observability cell)...
[INFO] ℹ️ Monitoring can be configured via:
[INFO] • Local postgres_exporter (step 04-monitoring-setup.sh)
[INFO] • Observability Cell integration (step 02-observability-cell)
[ OK ] ✅ postgres_exporter is installed
[INFO] Version: 0.10.1-1ubuntu0.22.04.3
[ OK ] ✅ postgres_exporter-web-universe-main-dev.service is running
[WARN] ⚠️ Metrics endpoint not responding
[INFO] ℹ️ Monitoring user 'postgres_exporter' not found in PostgreSQL
[INFO] This is expected if using observability cell remote monitoring
[INFO] ℹ️ Monitoring check passed (local or observability cell)
[INFO] 🔍 Checking Backup Setup (pgBackRest + WAL archiving)...
[ OK ] ✅ pgBackRest is installed
[INFO] Version: pgBackRest 2.56.0
[ OK ] ✅ WAL archiving is enabled (archive_mode=on)
[ OK ] ✅ archive_command is configured for pgBackRest
[INFO] Command: timeout 30 /usr/bin/pgbackrest --stanza=web-universe-main-dev-coordinator archive-push %p
[ OK ] ✅ pgBackRest configuration exists
[ OK ] ✅ pgBackRest stanza 'web-universe-main-dev-coordinator' is initialized
[ OK ] ✅ Backups exist (4 full backup(s))
[INFO] Latest backup info:
[ OK ] ✅ Automated backup cron jobs are configured
[INFO] Schedule:
0 2 * * 0 root /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
0 2 * * 1-6 root /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[ OK ] ✅ Backup directory exists: /var/lib/pgbackrest
[INFO] Total backup size: 6.1G
[INFO] 🔍 Checking Worker Backup Coverage...
[INFO] ℹ️ Worker backups are optional for development environments
[INFO] For production, ensure all workers have backup coverage
[INFO] Checking worker 1/1: worker-01...
[WARN] ⚠️ Worker worker-01 stanza exists but status unknown
[INFO] ℹ️ Incomplete worker backup coverage (0/1) - OK for dev
[INFO] 🔍 Checking Synchronous Replication (RPO=0)...
[INFO] ℹ️ Synchronous replication (RPO=0) is optional for development
[INFO] For production with zero data loss requirement, enable sync replication
[INFO] ℹ️ Worker worker-01 synchronous replication NOT configured
[INFO] └─ synchronous_commit: on
[INFO] └─ synchronous_standby_names:
[ OK ] ✅ All workers have synchronous replication (RPO=0)
[INFO] 🔍 Checking Connection and Memory Optimization...
[ OK ] ✅ Coordinator max_connections optimized: 150
[ OK ] ✅ Coordinator work_mem optimized: 8MB
[ OK ] ✅ Worker worker-01 max_connections optimized: 100
[ OK ] ✅ Worker worker-01 work_mem optimized: 8MB
[ OK ] ✅ All instances have optimized connection and memory settings
[INFO] 🔍 Checking Optimizations...
[ OK ] ✅ Citus coordinator host configured: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[WARN] ⚠️ Periodic integrity checks NOT configured
[INFO] Configure with: ./setup/04-postgresql/steps/04-production-optimizations.sh
[WARN] ⚠️ Backup schedule NOT staggered (all at :00)
[INFO] Optimize with: ./setup/04-postgresql/steps/04-production-optimizations.sh
[2026-02-05 07:23:58 UTC] USER=www-data EUID=0 PID=183506 ACTION=fsop ARGS=test -f /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 07:23:58 UTC] USER=www-data EUID=0 PID=183516 ACTION=fsop ARGS=grep -q ## Cipher Key Management /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[WARN] ⚠️ Cipher key management documentation missing
[INFO] Add with: ./setup/04-postgresql/steps/04-production-optimizations.sh
[INFO] ℹ️ Offsite backup (repo2) not configured (optional for production)
[INFO] Setup guide: ./setup/05-db/engine/postgresql/steps/14-setup-offsite-backup.sh
[WARN] ⚠️ Some production optimizations incomplete
[INFO] 🔍 Checking Citus Maintenance Daemon Health...
[INFO] Checking for stuck Citus Maintenance Daemons...
[ OK ] ✅ Citus Maintenance Daemons are healthy
[INFO] Checking for stuck distributed table operations...
[ OK ] ✅ No stuck distributed table operations
[INFO] Testing distributed table operations (10s timeout)...
[WARN] ⚠️ CRITICAL: Distributed table test TIMED OUT (10s)
[WARN] Citus cluster is NOT operational - distributed tables cannot be created
[WARN] This confirms maintenance daemons are stuck
[WARN]
[WARN] 🔧 ACTION REQUIRED: Restart coordinator before using Citus
[WARN] sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-coordinator.service
[INFO] Checking for lock contention...
[ OK ] ✅ No lock contention detected
[INFO] Checking for lingering prepared transactions...
[ OK ] ✅ No lingering prepared transactions
[WARN] ⚠️ Citus cluster has health issues - see warnings above
[WARN]
[WARN] ⚡ IMMEDIATE ACTION: Restart coordinator to restore Citus functionality
[WARN] sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev.service
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📋 PRODUCTION READINESS SUMMARY
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Component Status Production Ready?
───────────────────────── ─────────────── ────────────────────
Citus Cluster ✅ Operational YES
High Availability ✅ Configured YES
SSL/TLS Security ✅ Enabled YES
PgBouncer ✅ Running YES
Monitoring ✅ Operational YES
Backups (Coordinator) ✅ Configured YES
Backups (Workers) ✅ Configured YES
Sync Replication (RPO=0) ✅ Enabled YES
Connection Optimization ✅ Configured YES
Optimizations ⚠️ Incomplete OPTIONAL
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] 🎉 PRODUCTION READY: 100% (3/3 critical checks passed)
[ OK ] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ All critical components are operational and production-ready
[INFO] Next steps:
[INFO] 1. Configure Prometheus to scrape metrics: http://localhost:9190/metrics
[INFO] 2. Import Grafana dashboards for PostgreSQL + Citus monitoring
[INFO] 3. Setup alerting rules for critical metrics
[INFO] 4. Schedule regular restore drills (monthly)
[INFO] 5. Review /var/www/html/skeleton.dev.fastorder.com/fixing/scripts/PRODUCTION_READINESS.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✓ Verification process completed successfully
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 18-production-optimizations.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m PostgreSQL Production Optimizations
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Enable Sync Replication: --auto
[0;34m[INFO][0m 1️⃣ Configuring Citus coordinator hostname...
[0;32m[ OK ][0m ✅ Coordinator hostname already configured: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[0;34m[INFO][0m 2️⃣ Configuring synchronous replication for RPO=0...
[0;34m[INFO][0m Synchronous replication NOT enabled (use './04-production-optimizations.sh yes' to enable)
[0;34m[INFO][0m Current configuration: async replication (RPO > 0)
[0;34m[INFO][0m
[0;34m[INFO][0m To enable safely after deployment:
[0;34m[INFO][0m /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/enable_sync_replication_safe.sh \
[0;34m[INFO][0m /var/run/postgresql-web-universe-main-dev-worker-01 worker_01_standby_01
[0;34m[INFO][0m 3️⃣ Adding periodic integrity check cron jobs...
[2026-02-05 07:24:01 UTC] USER=www-data EUID=0 PID=183772 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-integrity-web-universe-main-dev
[0;32m[ OK ][0m ✅ Integrity check cron jobs configured
[0;34m[INFO][0m Daily checks: 02:15, 03:30, 04:45 (coordinator, worker-01, worker-02)
[0;34m[INFO][0m Weekly verify: Sundays at same times
[0;34m[INFO][0m 4️⃣ Updating backup schedule with staggered timing...
[0;32m[ OK ][0m ✅ Backup schedule staggered:
[0;34m[INFO][0m Coordinator: 02:05 (full: Sun, diff: Mon-Sat)
[0;34m[INFO][0m Worker-01: 03:10 (full: Sun, diff: Mon-Sat)
[0;34m[INFO][0m Worker-02: 04:15 (full: Sun, diff: Mon-Sat)
[0;34m[INFO][0m 5️⃣ Documenting cipher key backup procedures...
[2026-02-05 07:24:01 UTC] USER=www-data EUID=0 PID=183811 ACTION=fsop ARGS=test -f /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
Sorry, user www-data is not allowed to execute '/usr/bin/grep -q ## Cipher Key Management /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md' as root on web-03.
[0;32m[ OK ][0m ✅ Cipher key documentation added to /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[0;34m[INFO][0m 6️⃣ Checking offsite backup configuration...
[0;34m[INFO][0m ℹ️ Offsite backup (repo2) is NOT configured
[0;34m[INFO][0m Configuration example: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[0;34m[INFO][0m Setup instructions: ./setup/04-postgresql/steps/08-setup-offsite-backup.sh
[0;32m[ OK ][0m ✅ Offsite backup example available: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Production Optimizations Complete
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[ OK ][0m Applied optimizations:
[0;32m[ OK ][0m 1. ✅ Citus coordinator hostname
[0;32m[ OK ][0m 2. ⏭️ Synchronous replication (RPO=0)
[0;32m[ OK ][0m 3. ✅ Periodic integrity checks (daily + weekly)
[0;32m[ OK ][0m 4. ✅ Staggered backup schedule (reduced load spikes)
[0;32m[ OK ][0m 5. ✅ Cipher key backup documentation
[0;32m[ OK ][0m 6. ✅ Offsite backup (repo2) example configuration
[0;34m[INFO][0m Next steps:
[0;34m[INFO][0m 1. Backup cipher keys to secure vault immediately
[0;34m[INFO][0m 2. Set up S3/MinIO for offsite backups:
[0;34m[INFO][0m - Instructions: ./setup/04-postgresql/steps/08-setup-offsite-backup.sh
[0;34m[INFO][0m - Example config: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[0;34m[INFO][0m 3. Configure alerting for backup failures (cron exit codes)
[0;34m[INFO][0m 4. Test restore drill from offsite repository
[0;34m[INFO][0m 5. Enable RPO=0 if needed: ./04-production-optimizations.sh yes
[0;32m[ OK ][0m System is now production-grade! 🎉
[0;32m✓[0m ✔ PostgreSQL creation completed
[0;34m[INFO][0m Cleaning up temporary files...
[INFO] Starting cleanup of temporary files...
[INFO] Cleaning up SSL temp files for web-universe-main-dev...
[INFO] Cleaning up old provisioning logs...
[INFO] Cleaning up old configuration backups...
[0;32m✓[0m ✔ Cleanup completed
[0;32m✓[0m ✅ Database infrastructure (postgresql) setup completed successfully
[0;34m[INFO][0m Using environment from web interface: web-universe-main-dev
[0;32m[2026-02-05 07:24:02][0m Using web-provided environment: web-universe-main-dev
[0;32m[2026-02-05 07:24:02][0m Service: web, Zone: universe, Branch: main, Env: dev
[0;32m✓[0m Environment initialized successfully (mode: general)
[0;34m[INFO][0m Starting finalizing setup process...
[0;34m[INFO][0m Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Found 3 step(s) to execute
[0;34m[INFO][0m 📦 Step 1/3: enable_disable_all_applications...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
════════════════════════════════════════════════════════════════════════════════
Environment Services Management
════════════════════════════════════════════════════════════════════════════════
Environment: web-universe-main-dev
Action: enable
Triggered by: false
════════════════════════════════════════════════════════════════════════════════
🔍 Scanning for environment-specific services...
✅ Found 5 services for environment: web-universe-main-dev
📋 Services to enable:
────────────────────────────────────────────────────────────────────────────────
• confluent-connect-web-universe-main-dev_coordinator.service [active/unmasked/enabled]
• confluent-kraft-web-universe-main-dev_coordinator.service [active/unmasked/enabled]
• pgbouncer@web-universe-main-dev.service [active/unmasked/enabled]
• postgresql@web-universe-main-dev-coordinator.service [active/unmasked/enabled]
• postgresql@web-universe-main-dev-worker-01.service [active/unmasked/enabled]
────────────────────────────────────────────────────────────────────────────────
❌ Cancelled by user
[0;32m[OK][0m ✅ Step 1 completed: 01-enable_disable_all_applications.sh
[0;34m[INFO][0m 📦 Step 2/3: verify monitoring...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Monitoring Verification for web-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1️⃣ Detecting installed services...
Failed to print table: Broken pipe
[0;32m[OK][0m ✓ PostgreSQL detected
Failed to print table: Broken pipe
[0;32m[OK][0m ✓ Kafka detected
Failed to print table: Broken pipe
[0;32m[OK][0m ✓ PgBouncer detected
[INFO] Services to verify: postgresql kafka pgbouncer
[INFO] 2️⃣ Verifying exporters are running...
[0;32m[OK][0m ✓ PostgreSQL exporter is running
[0;32m[OK][0m ✓ Kafka JMX exporter is running
[0;32m[OK][0m ✓ PgBouncer exporter is running
[INFO] 3️⃣ Verifying Prometheus configuration...
[2026-02-05 07:24:04 UTC] USER=www-data EUID=0 PID=184201 ACTION=passthru ARGS=grep -q job_name: 'postgresql' /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
[0;32m[OK][0m ✓ postgresql is configured in Prometheus
[2026-02-05 07:24:04 UTC] USER=www-data EUID=0 PID=184222 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
[WARN] ⚠️ kafka is not configured in Prometheus scrape targets
[2026-02-05 07:24:04 UTC] USER=www-data EUID=0 PID=184243 ACTION=passthru ARGS=grep -q job_name: 'pgbouncer' /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
[WARN] ⚠️ pgbouncer is not configured in Prometheus scrape targets
[INFO] 4️⃣ Verifying Prometheus is actively scraping...
[0;32m[OK][0m ✓ Prometheus is running
[0;32m[OK][0m ✓ postgresql target is UP in Prometheus
[WARN] ⚠️ kafka target is not UP in Prometheus (may still be initializing)
[WARN] ⚠️ pgbouncer target is not UP in Prometheus (may still be initializing)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Monitoring Verification Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[WARN] Some monitoring issues were detected:
[WARN] Prometheus Configuration Issues:
- kafka not configured in Prometheus
- pgbouncer not configured in Prometheus
[WARN] Automatically running monitoring setup scripts to fix issues...
[INFO] Running Kafka monitoring setup...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Kafka Monitoring Integration for web-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-web-universe-main-dev
[1;32m[OK][0m Observability cell endpoints registered for web-universe-main-dev
[1;32m[OK][0m ✓ Observability cell is ready
[INFO] 2️⃣ Setting up Kafka JMX exporter integration...
[INFO] JMX Exporter port calculated for web-universe-main-dev: 9362 (offset: 54)
[INFO] Checking observability cell readiness: obs-web-universe-main-dev
[1;32m[OK][0m Observability cell endpoints registered for web-universe-main-dev
[INFO] Setting up Kafka JMX exporter for web-universe-main-dev
[INFO] JMX Prometheus Java Agent already exists at /opt/kafka/libs/jmx_prometheus_javaagent.jar
[2026-02-05 07:24:09 UTC] USER=www-data EUID=0 PID=184494 ACTION=passthru ARGS=mv /tmp/jmx_exporter.yml /opt/kafka/config/jmx_exporter.yml
[2026-02-05 07:24:09 UTC] USER=www-data EUID=0 PID=184503 ACTION=passthru ARGS=chmod 644 /opt/kafka/config/jmx_exporter.yml
[1;32m[OK][0m JMX exporter configuration created at /opt/kafka/config/jmx_exporter.yml
[1;32m[OK][0m JMX exporter configuration created
[INFO] Configuring Kafka systemd services to use JMX exporter...
[2026-02-05 07:24:09 UTC] USER=www-data EUID=0 PID=184527 ACTION=fsop ARGS=test -f /etc/systemd/system/[2026-02-05
[INFO] All Kafka services already configured with JMX exporter
[1;32m[OK][0m Kafka JMX exporter integration complete
[INFO] Metrics endpoint: http://142.93.238.16:9362/metrics
[INFO] Prometheus will automatically scrape: https://metrics-web-universe-main-dev.fastorder.com:9090
[INFO] View dashboards at: https://dashboards-web-universe-main-dev.fastorder.com
[1;32m[OK][0m ✓ Kafka JMX exporter integration complete
[INFO] Configuring KAFKA_OPTS environment variable for kafka user...
[2026-02-05 07:24:10 UTC] USER=www-data EUID=0 PID=184548 ACTION=passthru ARGS=grep -q KAFKA_OPTS.*javaagent.*jmx_prometheus_javaagent.*=9362: /home/kafka/.bashrc
[2026-02-05 07:24:10 UTC] USER=www-data EUID=0 PID=184569 ACTION=passthru ARGS=sed -i /export KAFKA_OPTS=.*jmx_prometheus_javaagent/d /home/kafka/.bashrc
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[1;32m[OK][0m ✓ KAFKA_OPTS configured in /home/kafka/.bashrc (port 9362)
[INFO] 2.5️⃣ Enabling JMX exporter in Kafka systemd service...
[2026-02-05 07:24:10 UTC] USER=www-data EUID=0 PID=184603 ACTION=passthru ARGS=grep -q javaagent.*jmx_prometheus_javaagent /etc/systemd/system/confluent-kraft-web-universe-main-dev_coordinator.service
[1;32m[OK][0m ✓ JMX exporter already enabled in Kafka systemd services
[INFO] 2.6️⃣ Configuring Prometheus to scrape Kafka metrics...
[2026-02-05 07:24:10 UTC] USER=www-data EUID=0 PID=184624 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
[INFO] Adding Kafka scrape target to Prometheus configuration...
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[2026-02-05 07:24:10 UTC] USER=www-data EUID=0 PID=184657 ACTION=passthru ARGS=sed -i /# Prometheus self-monitoring/r /tmp/prometheus_kafka_add.yml /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
[ERROR] Invalid Prometheus configuration - rolling back
[2026-02-05 07:24:10 UTC] USER=www-data EUID=0 PID=184690 ACTION=passthru ARGS=sed -i /job_name: 'kafka'/,+6d /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
[2026-02-05 07:24:10 UTC] USER=www-data EUID=0 PID=184712 ACTION=fsop ARGS=rm -f /tmp/prometheus_kafka_add.yml
[INFO] 3️⃣ Registering Kafka nodes to monitoring database...
[INFO] Detected Kafka version: 3.9.1
[INFO] Registering Kafka Broker to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Kafka Broker
[INFO] Identifier: web-universe-main-dev-broker-01
[INFO] Identifier Parent: cluster
[INFO] IP: 142.93.238.16
[INFO] Port: 9092
[INFO] FQDN: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 0fe22eef-a876-408e-9099-f79ee8d192b7
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[1;32m[OK][0m ✓ Kafka broker registered
[INFO] Registering Kafka Connect to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Kafka Connect
[INFO] Identifier: web-universe-main-dev-connect-01
[INFO] Identifier Parent: cluster
[INFO] IP: 142.93.238.16
[INFO] Port: 8083
[INFO] FQDN: eventbus-web-universe-main-dev-kafka-connect.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 269c6f12-e045-4268-8bc4-73c5e936d212
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[1;32m[OK][0m ✓ Kafka Connect registered
[INFO] Schema Registry not running, skipping registration
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Kafka Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Metrics: http://localhost:9362/metrics
[INFO] Prometheus: https://metrics-web-universe-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-web-universe-main-dev.fastorder.com
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✓ Kafka monitoring setup completed
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Step 2 completed: 02-verify-monitoring.sh
[0;34m[INFO][0m 📦 Step 3/3: register backup infrastructure...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔧 Registering Core Services & Backup Infrastructure for web-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1️⃣ Registering Main App...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Main App
[INFO] Identifier: web-universe-main-dev-main-app
[INFO] Identifier Parent: application
[INFO] IP: 142.93.238.16
[INFO] Port: 8080
[INFO] FQDN: app-web-universe-main-dev.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 0ec21f30-1acd-45a5-a2b2-98031576cc92
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
/opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps/03-register-backup-infrastructure.sh: line 70: ok: command not found
[INFO] 2️⃣ Registering Audit Service...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Audit Service
[INFO] Identifier: web-universe-main-dev-audit
[INFO] Identifier Parent: application
[INFO] IP: 142.93.238.16
[INFO] Port: 8081
[INFO] FQDN: audit-web-universe-main-dev.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 2b79b583-04b8-4ab2-892d-219b4aa3533a
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
/opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps/03-register-backup-infrastructure.sh: line 85: ok: command not found
[INFO] 3️⃣ Registering PostgreSQL Backup Node...
[ERROR] Invalid identifier format: backup-db
[ERROR] Expected formats:
[ERROR] SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR] iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR] obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️ Failed to register PostgreSQL backup node (non-blocking)
[INFO] 4️⃣ Registering Elasticsearch Backup Node...
[ERROR] Invalid identifier format: backup-search
[ERROR] Expected formats:
[ERROR] SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR] iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR] obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️ Failed to register Elasticsearch backup node (non-blocking)
[INFO] 5️⃣ Registering Kafka Backup Node...
[ERROR] Invalid identifier format: backup-eventbus
[ERROR] Expected formats:
[ERROR] SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR] iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR] obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️ Failed to register Kafka backup node (non-blocking)
[INFO] 6️⃣ Registering Backup Orchestrator...
[ERROR] Invalid identifier format: backup-orchestrator
[ERROR] Expected formats:
[ERROR] SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR] iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR] obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️ Failed to register Backup orchestrator (non-blocking)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Core Services & Backup Infrastructure Registration Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Registered core services:
[INFO] 🚀 main-app → Core application service
[INFO] 📋 audit → Centralized audit logging (WORM)
[INFO] Registered backup nodes:
[INFO] 📦 backup-db → PostgreSQL backup (pgBackRest, PITR)
[INFO] 📦 backup-search → Elasticsearch snapshots (ILM, S3)
[INFO] 📦 backup-eventbus → Kafka log segments (replication)
[INFO] 📦 backup-orchestrator → Central backup coordination
[INFO] Dashboard: https://skeleton.dev.fastorder.com/dashboard/monitoring
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Step 3 completed: 03-register-backup-infrastructure.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ finalizing setup completed successfully!
[0;32m[OK][0m Executed all 3 steps
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Service: web
[0;34m[INFO][0m Zone: universe
[0;34m[INFO][0m Branch: main
[0;34m[INFO][0m Env: dev