📊 Provisioning Job Status

Environment: Web Universe Main Dev on web-03

❌ Failed

⏱️ Timing Summary

🕐
Requested 2026-02-05 07:11:02 2 days ago
▶️
Started 2026-02-05 07:11:02 2 days ago
🏁
Finished 2026-02-05 07:16:02 2 days ago
⏲️
Total Duration 5 minutes

📋 Job Details

Job ID: ddd09ec2-cb6e-4c2d-83a4-2fe9cc94a9e7
Action: SETUP
Status: ❌ FAILED
Environment: web-universe-main-dev
Resource: web-03 (Provider)
Requested By: admin
Parameters: 
"{\"env\": \"dev\", \"zone\": \"universe\", \"branch\": \"main\", \"db_app\": \"postgresql\", \"service\": \"web\", \"db_enabled\": true, \"pg_standby\": 0, \"pg_workers\": 1, \"description\": \"\", \"iam_enabled\": false, \"worker_1_ip\": \"\", \"eventbus_app\": \"kafka\", \"worker_1_fqdn\": \"db-web-universe-main-dev-postgresql-worker-01.fastorder.com\", \"search_enabled\": false, \"eventbus_enabled\": true, \"postgresql_enabled\": true, \"postgresql_run_verification\": true}"
❌ Error: One or more steps failed. Check run logs for details.
⚠️ Job Failed

This job encountered an error. You can restart from the failed step.

📢 Viewing Old Job Attempt

This job has been restarted. You are viewing an older attempt. The logs and status shown below are from the latest retry.

🔄 View Latest Attempt Latest Status:

🔄 Resume & Restart Options

This job failed at one of the steps below. You can resume from where it failed to save time and avoid re-running successful steps.

💡
1 step failed

📝 Execution Steps (9)

0/9 completed 1 failed
0% (0/9 steps)
1
00-preflight-checks local
⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars) 
Loading logs...
2
00-terraform-provision local
⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars) 
Loading logs...
3
01-prepare-environment local
⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars) 
Loading logs...
4
02-iam local
⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars) 
Loading logs...
5
02-observability-cell local
⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars) 
Loading logs...
6
03-search local
⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars) 
Loading logs...
7
04-eventbus local
⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars) 
Loading logs...
8
05-db local
❌ FAILED
⏰ Started: 2026-02-05 07:11:02
🏁 Finished: 2026-02-05 07:16:02
⏱️ Duration: 5 minutes
📄 View Logs (398702 chars) 
[INFO] Using database engine from DB_ENGINE environment variable: postgresql
[INFO] Cleaning up any existing locks...

Starting database engine: postgresql
═══════════════════════════════════════════════

[INFO] Using environment from web interface: web-universe-main-dev
[2026-02-05 07:11:02] Using web-provided environment: web-universe-main-dev
[2026-02-05 07:11:02] Service: web, Zone: universe, Branch: main, Env: dev
✓ Environment initialized successfully (mode: general)
[INFO] Checking observability cell readiness: obs-web-universe-main-dev
[OK]   Observability cell endpoints registered for web-universe-main-dev
[INFO] Observability cell verified for web-universe-main-dev
[INFO] Monitoring will be configured after PostgreSQL deployment (step 10-monitoring-setup.sh)
[INFO] Citus mode ENABLED
[INFO] → Coordinator + 1 worker(s) + 0 standby node(s) per worker
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up coordinator (Citus control plane)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-02-05 07:11:03 UTC] USER=unknown EUID=33 PID=112870 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 07:11:03 UTC] USER=unknown EUID=33 PID=112877 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 07:11:03 UTC] USER=unknown EUID=33 PID=112884 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-02-05 07:11:03 UTC] USER=unknown EUID=33 PID=112891 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 07:11:03 UTC] USER=unknown EUID=33 PID=112898 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-02-05 07:11:03 UTC] USER=unknown EUID=33 PID=112905 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for web in universe-dev...
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=81c80b1b-f9eb-44f5-925f-0455b7853b9d JOB_UUID=ddd09ec2-cb6e-4c2d-83a4-2fe9cc94a9e7

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=81c80b1b-f9eb-44f5-925f-0455b7853b9d)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: coordinator
[INFO] PostgreSQL IP: 10.100.1.54
[INFO] Primary hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com

[INFO] Adding /etc/hosts entries for coordinator...
[INFO]   1. db-web-universe-main-dev-postgresql.fastorder.com → 10.100.1.54 (primary/short)
[INFO]   2. db-web-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.54 (compatibility)

[INFO]   ✅ db-web-universe-main-dev-postgresql.fastorder.com already exists with correct IP
[INFO]   ✅ db-web-universe-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.54    db-web-universe-main-dev-postgresql.fastorder.com
  10.100.1.54    db-web-universe-main-dev-postgresql-coordinator.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=81c80b1b-f9eb-44f5-925f-0455b7853b9d)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-web-universe-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      web-universe-main-dev.fastorder.com
  Internal IP: 10.100.1.54 (from topology: db-coordinator-postgresql)
  Shared IP:   142.93.238.16 (legacy/public, also included in SANs)
  Coordinator variants:
    - db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 07:11:06 UTC] USER=www-data EUID=0 PID=113088 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator and /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:06 UTC] USER=www-data EUID=0 PID=113097 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-02-05 07:11:06 UTC] USER=www-data EUID=0 PID=113107 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-113051
[2026-02-05 07:11:06 UTC] USER=www-data EUID=0 PID=113116 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-113051/ra_root.crt
[2026-02-05 07:11:06 UTC] USER=www-data EUID=0 PID=113125 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-113051/ra_root.key
[2026-02-05 07:11:06 UTC] USER=www-data EUID=0 PID=113134 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-113051/ra_root.crt
[2026-02-05 07:11:06 UTC] USER=www-data EUID=0 PID=113143 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-113051/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[2026-02-05 07:11:08 UTC] USER=www-data EUID=0 PID=113234 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-113051/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 07:11:09 UTC] USER=www-data EUID=0 PID=113243 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-113051/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 07:11:09 UTC] USER=www-data EUID=0 PID=113252 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-02-05 07:11:09 UTC] USER=www-data EUID=0 PID=113261 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-113051/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:09 UTC] USER=www-data EUID=0 PID=113270 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:09 UTC] USER=www-data EUID=0 PID=113279 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:09 UTC] USER=www-data EUID=0 PID=113289 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 07:11:09 UTC] USER=www-data EUID=0 PID=113300 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 07:11:09 UTC] USER=www-data EUID=0 PID=113309 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 07:11:09 UTC] USER=www-data EUID=0 PID=113325 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 07:11:09 UTC] USER=www-data EUID=0 PID=113334 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 07:11:09 UTC] USER=www-data EUID=0 PID=113343 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:09 UTC] USER=www-data EUID=0 PID=113358 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-web-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:db-web-universe-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-web-universe-main-dev-postgresql.fastorder.com, IP Address:10.100.1.54, IP Address:127.0.0.1, IP Address:142.93.238.16
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: web-universe-main-dev
Node:        coordinator
Primary CN:  db-web-universe-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-web-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:11:09 UTC] USER=www-data EUID=0 PID=113423 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-02-05 07:11:09 UTC] USER=www-data EUID=0 PID=113436 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 07:11:09 UTC] USER=www-data EUID=0 PID=113459 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-02-05 07:11:09 UTC] USER=www-data EUID=0 PID=113470 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 07:11:09 UTC] USER=www-data EUID=0 PID=113490 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:10 UTC] USER=www-data EUID=0 PID=113524 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:10 UTC] USER=www-data EUID=0 PID=113533 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:10 UTC] USER=www-data EUID=0 PID=113543 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:11:10 UTC] USER=www-data EUID=0 PID=113563 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:11:10 UTC] USER=www-data EUID=0 PID=113578 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:10 UTC] USER=www-data EUID=0 PID=113599 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:11:10 UTC] USER=www-data EUID=0 PID=113610 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:11:10 UTC] USER=www-data EUID=0 PID=113628 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:11:10 UTC] USER=www-data EUID=0 PID=113651 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:11:10 UTC] USER=www-data EUID=0 PID=113682 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:11:10 UTC] USER=www-data EUID=0 PID=113691 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:11:10 UTC] USER=www-data EUID=0 PID=113700 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:11:10 UTC] USER=www-data EUID=0 PID=113709 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:10 UTC] USER=www-data EUID=0 PID=113765 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:10 UTC] USER=www-data EUID=0 PID=113774 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:10 UTC] USER=www-data EUID=0 PID=113792 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113801 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113810 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113819 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113828 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113837 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113846 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113855 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113865 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113875 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113885 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113894 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113903 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113912 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113921 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113930 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113939 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113948 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113957 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113966 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=113975 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=114000 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=114012 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=114021 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=114030 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=114039 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=114048 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=114057 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=114066 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=114075 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:11 UTC] USER=www-data EUID=0 PID=114086 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:11:12 UTC] USER=www-data EUID=0 PID=114095 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:11:12 UTC] USER=www-data EUID=0 PID=114104 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:11:12 UTC] USER=www-data EUID=0 PID=114113 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:11:12 UTC] USER=www-data EUID=0 PID=114125 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:12 UTC] USER=www-data EUID=0 PID=114144 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:12 UTC] USER=www-data EUID=0 PID=114154 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:12 UTC] USER=www-data EUID=0 PID=114165 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:12 UTC] USER=www-data EUID=0 PID=114176 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:12 UTC] USER=www-data EUID=0 PID=114185 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:11:12 UTC] USER=www-data EUID=0 PID=114199 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:11:12 UTC] USER=www-data EUID=0 PID=114208 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:12 UTC] USER=www-data EUID=0 PID=114218 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:11:12 UTC] USER=www-data EUID=0 PID=114228 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:11:12 UTC] USER=www-data EUID=0 PID=114237 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:11:12 UTC] USER=www-data EUID=0 PID=114246 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:11:12 UTC] USER=www-data EUID=0 PID=114256 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:11:12 UTC] USER=www-data EUID=0 PID=114269 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:11:13 UTC] USER=www-data EUID=0 PID=114279 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:11:13 UTC] USER=www-data EUID=0 PID=114288 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 07:11:13 UTC] USER=www-data EUID=0 PID=114297 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
   ✅ Symlinked client-key.pem
[2026-02-05 07:11:13 UTC] USER=www-data EUID=0 PID=114317 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:11:13 UTC] USER=www-data EUID=0 PID=114341 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: postgres
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:11:13 UTC] USER=www-data EUID=0 PID=114425 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-02-05 07:11:13 UTC] USER=www-data EUID=0 PID=114434 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 07:11:13 UTC] USER=www-data EUID=0 PID=114446 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-02-05 07:11:13 UTC] USER=www-data EUID=0 PID=114462 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 07:11:13 UTC] USER=www-data EUID=0 PID=114475 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:14 UTC] USER=www-data EUID=0 PID=114524 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:14 UTC] USER=www-data EUID=0 PID=114553 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:14 UTC] USER=www-data EUID=0 PID=114563 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:11:14 UTC] USER=www-data EUID=0 PID=114573 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:11:14 UTC] USER=www-data EUID=0 PID=114582 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:14 UTC] USER=www-data EUID=0 PID=114592 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:11:14 UTC] USER=www-data EUID=0 PID=114611 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:11:14 UTC] USER=www-data EUID=0 PID=114634 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:11:14 UTC] USER=www-data EUID=0 PID=114654 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:11:14 UTC] USER=www-data EUID=0 PID=114664 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:11:14 UTC] USER=www-data EUID=0 PID=114673 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:14 UTC] USER=www-data EUID=0 PID=114683 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:14 UTC] USER=www-data EUID=0 PID=114692 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:11:14 UTC] USER=www-data EUID=0 PID=114701 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114710 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114719 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114728 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114739 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114765 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114774 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114796 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114807 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114819 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114828 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114837 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114846 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114855 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114877 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114887 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114904 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114923 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114933 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114942 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114951 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114960 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114969 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114978 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=114989 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=115001 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=115010 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:11:15 UTC] USER=www-data EUID=0 PID=115019 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115029 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115039 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115050 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115059 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115069 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115080 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115089 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115098 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115107 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115116 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115125 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115134 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115143 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115152 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115165 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115175 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115184 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115193 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115202 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115211 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115220 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115229 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115238 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115247 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115256 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115265 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115274 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115284 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115303 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115315 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115324 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115333 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 07:11:16 UTC] USER=www-data EUID=0 PID=115342 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:11:17 UTC] USER=www-data EUID=0 PID=115351 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:11:17 UTC] USER=www-data EUID=0 PID=115364 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:11:17 UTC] USER=www-data EUID=0 PID=115382 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: postgres
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=81c80b1b-f9eb-44f5-925f-0455b7853b9d)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-coordinator-postgresql environment: db-web-universe-main-dev-postgresql-coordinator.fastorder.com (10.100.1.54)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.54
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: coordinator
[INFO] Data dir:   /data/postgresql/17/web-universe-main-dev/coordinator
[INFO] Port:       5432
[INFO] Hostname:   db-web-universe-main-dev-postgresql-coordinator
[2026-02-05 07:11:18 UTC] USER=www-data EUID=0 PID=115506 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:18 UTC] USER=www-data EUID=0 PID=115528 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:18 UTC] USER=www-data EUID=0 PID=115557 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:19 UTC] USER=www-data EUID=0 PID=115584 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-web-universe-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      web-universe-main-dev.fastorder.com
  Internal IP: 10.100.1.54 (from topology: db-coordinator-postgresql)
  Shared IP:   142.93.238.16 (legacy/public, also included in SANs)
  Coordinator variants:
    - db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 07:11:19 UTC] USER=www-data EUID=0 PID=115640 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator and /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:19 UTC] USER=www-data EUID=0 PID=115668 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-02-05 07:11:19 UTC] USER=www-data EUID=0 PID=115689 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-115591/ra_root.crt
[2026-02-05 07:11:19 UTC] USER=www-data EUID=0 PID=115698 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-115591/ra_root.key
[2026-02-05 07:11:19 UTC] USER=www-data EUID=0 PID=115707 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-115591/ra_root.crt
[2026-02-05 07:11:19 UTC] USER=www-data EUID=0 PID=115716 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-115591/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[2026-02-05 07:11:21 UTC] USER=www-data EUID=0 PID=115768 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-115591/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 07:11:21 UTC] USER=www-data EUID=0 PID=115787 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-115591/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 07:11:21 UTC] USER=www-data EUID=0 PID=115800 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-02-05 07:11:21 UTC] USER=www-data EUID=0 PID=115809 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-115591/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:21 UTC] USER=www-data EUID=0 PID=115818 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:21 UTC] USER=www-data EUID=0 PID=115827 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:21 UTC] USER=www-data EUID=0 PID=115836 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 07:11:21 UTC] USER=www-data EUID=0 PID=115847 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 07:11:21 UTC] USER=www-data EUID=0 PID=115856 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 07:11:21 UTC] USER=www-data EUID=0 PID=115866 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 07:11:21 UTC] USER=www-data EUID=0 PID=115875 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 07:11:21 UTC] USER=www-data EUID=0 PID=115885 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:21 UTC] USER=www-data EUID=0 PID=115895 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-web-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:db-web-universe-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-web-universe-main-dev-postgresql.fastorder.com, IP Address:10.100.1.54, IP Address:127.0.0.1, IP Address:142.93.238.16
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: web-universe-main-dev
Node:        coordinator
Primary CN:  db-web-universe-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-web-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-02-05 07:11:21 UTC] USER=www-data EUID=0 PID=115934 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 07:11:22 UTC] USER=www-data EUID=0 PID=115943 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-02-05 07:11:22 UTC] USER=www-data EUID=0 PID=115968 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 07:11:22 UTC] USER=www-data EUID=0 PID=116001 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-web-universe-main-dev-coordinator
[2026-02-05 07:11:22 UTC] USER=www-data EUID=0 PID=116035 ACTION=fsop ARGS=rm -rf /var/run/postgresql-web-universe-main-dev-coordinator
[OK]   No conflicting Postgres left on port 5432
[OK]   Using postgres password from vault provider
[2026-02-05 07:11:24 UTC] USER=www-data EUID=0 PID=116133 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.BOylB2
[2026-02-05 07:11:24 UTC] USER=www-data EUID=0 PID=116164 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.BOylB2
[2026-02-05 07:11:24 UTC] USER=www-data EUID=0 PID=116259 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev
[2026-02-05 07:11:24 UTC] USER=www-data EUID=0 PID=116290 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev
[2026-02-05 07:11:24 UTC] USER=www-data EUID=0 PID=116313 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/web-universe-main-dev
[INFO] Initializing cluster in /data/postgresql/17/web-universe-main-dev/coordinator (SCRAM; pwfile)
[WARN] Removing existing data directory: /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 07:11:24 UTC] USER=www-data EUID=0 PID=116339 ACTION=fsop ARGS=rm -rf /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 07:11:25 UTC] USER=www-data EUID=0 PID=116393 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 07:11:25 UTC] USER=www-data EUID=0 PID=116427 ACTION=fsop ARGS=chmod 700 /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 07:11:25 UTC] USER=www-data EUID=0 PID=116453 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-web-universe-main-dev-coordinator
[2026-02-05 07:11:25 UTC] USER=www-data EUID=0 PID=116483 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-web-universe-main-dev-coordinator
[2026-02-05 07:11:26 UTC] USER=www-data EUID=0 PID=116504 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-web-universe-main-dev-coordinator
[2026-02-05 07:11:26 UTC] USER=www-data EUID=0 PID=116513 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/web-universe-main-dev/coordinator --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.BOylB2
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /data/postgresql/17/web-universe-main-dev/coordinator ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/web-universe-main-dev/coordinator -l logfile start

[OK]   initdb complete
[2026-02-05 07:11:27 UTC] USER=www-data EUID=0 PID=116587 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.BOylB2
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-02-05 07:11:27 UTC] USER=www-data EUID=0 PID=116634 ACTION=fsop ARGS=cp /tmp/tmp.WcBRxTrMQE /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
[2026-02-05 07:11:27 UTC] USER=www-data EUID=0 PID=116685 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 07:11:28 UTC] USER=www-data EUID=0 PID=116710 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.Z9321i /etc/systemd/system/postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 07:11:28 UTC] USER=www-data EUID=0 PID=116734 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@web-universe-main-dev-coordinator.service
[OK]   systemd unit written
[2026-02-05 07:11:28 UTC] USER=www-data EUID=0 PID=116757 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 07:11:28 UTC] USER=www-data EUID=0 PID=116778 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 07:11:28 UTC] USER=www-data EUID=0 PID=116811 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-02-05 07:11:29 UTC] USER=www-data EUID=0 PID=116970 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-02-05 07:11:30 UTC] USER=www-data EUID=0 PID=117038 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Provisioning application database and Debezium role (if not exists)...
[INFO] Checking if database fastorder_web_universe_main_dev_db exists...
[INFO] DB check result: exit_code=0, output='[2026-02-05 07:11:31 UTC] USER=www-data EUID=0 PID=117204 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_web_universe_main_dev_db''
[INFO] Creating database fastorder_web_universe_main_dev_db...
[2026-02-05 07:11:31 UTC] USER=www-data EUID=0 PID=117227 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_web_universe_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[OK]   Database fastorder_web_universe_main_dev_db created
[INFO] Checking if role debezium_user exists...
[INFO] Role check result: exit_code=0, output='[2026-02-05 07:11:31 UTC] USER=www-data EUID=0 PID=117251 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[INFO] Creating role debezium_user...
[2026-02-05 07:11:31 UTC] USER=www-data EUID=0 PID=117278 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'qLWRD8yJBpKiCSjOeROu7zMR';
CREATE ROLE
[OK]   Role debezium_user created
[2026-02-05 07:11:31 UTC] USER=www-data EUID=0 PID=117301 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_web_universe_main_dev_db" TO debezium_user;
GRANT
[OK]   Application DB (fastorder_web_universe_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[INFO] Applying connection and memory optimizations...
[INFO] Current settings: max_connections=100, work_mem=4MB
[INFO] Target settings (coordinator): max_connections=150, work_mem=8MB
[2026-02-05 07:11:32 UTC] USER=www-data EUID=0 PID=117382 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 150;
ALTER SYSTEM
[2026-02-05 07:11:32 UTC] USER=www-data EUID=0 PID=117411 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-02-05 07:11:32 UTC] USER=www-data EUID=0 PID=117447 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   Settings applied to postgresql.auto.conf
[2026-02-05 07:11:33 UTC] USER=www-data EUID=0 PID=117465 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/coordinator/standby.signal
[INFO] Service recently started (3s ago) - restarting to apply max_connections...
[INFO] Stopping service...
[2026-02-05 07:11:33 UTC] USER=www-data EUID=0 PID=117487 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[INFO] Waiting for port 5432 to be released...
[OK]   Port 5432 released
[INFO] Starting service...
[2026-02-05 07:11:36 UTC] USER=www-data EUID=0 PID=117600 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 07:11:42 UTC] USER=www-data EUID=0 PID=117944 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[OK]   ✅ Optimization complete: max_connections=150, work_mem=8MB
[INFO] Setting postgres password via centralized script... for coordinator
[INFO] Temporarily disabling synchronous_commit on coordinator for password setting...
[OK]   Disabled synchronous_commit (was: on)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    web
  Zone:       universe
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator

Connection Info:
  Socket Dir: /var/run/postgresql-web-universe-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Initial setup: Using password from initdb
✓ PostgreSQL password already set during initdb
Storing password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/coordinator
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
[INFO] Restoring synchronous_commit on coordinator...
[OK]   Restored synchronous_commit to: on
[OK]   Password set and persisted
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: coordinator
[INFO] PostgreSQL IP: 10.100.1.54
[INFO] Primary hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com

[INFO] Adding /etc/hosts entries for coordinator...
[INFO]   1. db-web-universe-main-dev-postgresql.fastorder.com → 10.100.1.54 (primary/short)
[INFO]   2. db-web-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.54 (compatibility)

[INFO]   ✅ db-web-universe-main-dev-postgresql.fastorder.com already exists with correct IP
[INFO]   ✅ db-web-universe-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.54    db-web-universe-main-dev-postgresql.fastorder.com
  10.100.1.54    db-web-universe-main-dev-postgresql-coordinator.fastorder.com


[OK]   PostgreSQL 'web-universe-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key \
        host=db-web-universe-main-dev-postgresql-coordinator port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        web-universe-main-dev-postgresql-coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.54
[INFO]   Port:              5432
[INFO]   FQDN:              db-web-universe-main-dev-postgresql-coordinator
[INFO]   Status:            running
[INFO]   Environment:       web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 90feffd7-89fb-4afb-a63f-cc975d7e928c
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=81c80b1b-f9eb-44f5-925f-0455b7853b9d)
[INFO] 📦 03 role...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 07:11:52 UTC] USER=www-data EUID=0 PID=118639 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/coordinator/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    debezium_user
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   debezium_user
  Hostname:    db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118802 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-debezium_user
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118811 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118820 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-debezium_user/ra_root.key
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118829 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118838 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118853 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118862 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118872 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118881 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118890 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118899 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118908 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118917 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118926 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118935 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118947 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118967 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=118978 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=119012 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=119039 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=119058 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=119077 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=119089 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:53 UTC] USER=www-data EUID=0 PID=119117 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119126 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119135 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119144 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119153 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119162 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119171 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119180 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119189 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119199 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119217 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119227 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119246 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119264 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119273 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119282 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119292 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119301 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119310 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119320 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119330 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119339 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119352 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119362 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119371 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:54 UTC] USER=www-data EUID=0 PID=119380 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119389 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119401 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119412 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119425 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119443 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119454 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119464 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119473 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119483 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119514 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119523 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119532 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119541 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119561 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119570 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119580 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119589 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119600 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119611 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119627 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119638 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119648 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119657 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119666 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 07:11:55 UTC] USER=www-data EUID=0 PID=119675 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 07:11:56 UTC] USER=www-data EUID=0 PID=119684 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 07:11:56 UTC] USER=www-data EUID=0 PID=119693 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:11:56 UTC] USER=www-data EUID=0 PID=119712 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:11:56 UTC] USER=www-data EUID=0 PID=119721 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:11:56 UTC] USER=www-data EUID=0 PID=119730 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: debezium_user
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U debezium_user -d postgres

✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    web
  Zone:       universe
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd

Connection Info:
  Socket Dir: /var/run/postgresql-web-universe-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: fastorder_admin_gd
Storing password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: coordinator/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-web-universe-main-dev-postgresql.fastorder.com
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    fastorder_admin_gd
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   fastorder_admin_gd
  Hostname:    db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:12:06 UTC] USER=www-data EUID=0 PID=120374 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-fastorder_admin_gd
[2026-02-05 07:12:06 UTC] USER=www-data EUID=0 PID=120392 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
[2026-02-05 07:12:06 UTC] USER=www-data EUID=0 PID=120401 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-02-05 07:12:06 UTC] USER=www-data EUID=0 PID=120410 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:12:06 UTC] USER=www-data EUID=0 PID=120425 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:12:06 UTC] USER=www-data EUID=0 PID=120439 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:12:06 UTC] USER=www-data EUID=0 PID=120450 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 07:12:06 UTC] USER=www-data EUID=0 PID=120459 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 07:12:06 UTC] USER=www-data EUID=0 PID=120468 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:12:06 UTC] USER=www-data EUID=0 PID=120477 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:12:06 UTC] USER=www-data EUID=0 PID=120486 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:12:06 UTC] USER=www-data EUID=0 PID=120495 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120504 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120513 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120522 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120531 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120540 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120549 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120558 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120567 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120576 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120585 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120594 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120638 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120647 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120656 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120665 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120674 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120694 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120703 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120712 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120721 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120730 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120757 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:12:07 UTC] USER=www-data EUID=0 PID=120769 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=120778 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=120787 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=120796 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=120814 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=120840 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=120861 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=120885 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=120894 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=120904 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=120914 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=120923 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=120934 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=120954 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=120963 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=120972 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=120981 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=121000 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=121015 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:12:08 UTC] USER=www-data EUID=0 PID=121024 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 07:12:09 UTC] USER=www-data EUID=0 PID=121033 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:12:09 UTC] USER=www-data EUID=0 PID=121043 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 07:12:09 UTC] USER=www-data EUID=0 PID=121054 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 07:12:09 UTC] USER=www-data EUID=0 PID=121064 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:12:09 UTC] USER=www-data EUID=0 PID=121074 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:12:09 UTC] USER=www-data EUID=0 PID=121085 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:09 UTC] USER=www-data EUID=0 PID=121109 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:12:09 UTC] USER=www-data EUID=0 PID=121119 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 07:12:09 UTC] USER=www-data EUID=0 PID=121129 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 07:12:09 UTC] USER=www-data EUID=0 PID=121139 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 07:12:09 UTC] USER=www-data EUID=0 PID=121148 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 07:12:09 UTC] USER=www-data EUID=0 PID=121169 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:12:09 UTC] USER=www-data EUID=0 PID=121181 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 07:12:09 UTC] USER=www-data EUID=0 PID=121192 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 07:12:09 UTC] USER=www-data EUID=0 PID=121202 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:12:10 UTC] USER=www-data EUID=0 PID=121212 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:12:10 UTC] USER=www-data EUID=0 PID=121221 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:12:10 UTC] USER=www-data EUID=0 PID=121230 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 07:12:10 UTC] USER=www-data EUID=0 PID=121241 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 07:12:10 UTC] USER=www-data EUID=0 PID=121255 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 07:12:10 UTC] USER=www-data EUID=0 PID=121264 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:12:10 UTC] USER=www-data EUID=0 PID=121315 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: fastorder_admin_gd
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-web-universe-main-dev-coordinator:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_web_universe_main_dev_db already exists, skipping creation
[2026-02-05 07:12:11 UTC] USER=www-data EUID=0 PID=121408 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-02-05 07:12:11 UTC] USER=www-data EUID=0 PID=121461 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

=== Pre-flight Checks ===
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
✓ AWS Secrets Manager accessible

=== Retrieving Credentials from AWS ===
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials loaded for coordinator/fastorder_admin_gd: fastorder_admin_gd@db-web-universe-main-dev-postgresql.fastorder.com:5432/fastorder_web_universe_main_dev_db
✓ Credentials retrieved: fastorder_admin_gd@db-web-universe-main-dev-postgresql.fastorder.com:5432/fastorder_web_universe_main_dev_db
╔════════════════════════════════════════════╗
║  PostgreSQL Test Suite (AWS Secrets MGR)  ║
╚════════════════════════════════════════════╝

=== PostgreSQL Authentication Test ===
✗ PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-web-universe-main-dev-postgresql.fastorder.com" (10.100.1.54), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 07:12:17 UTC] USER=www-data EUID=0 PID=121739 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/coordinator/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : web-universe-main-dev
  IDENTIFIER  : coordinator
  PG HOST     : db-web-universe-main-dev-postgresql.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_web_universe_main_dev_db
  SCHEMA      : web
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
  DNS → 10.100.1.54
  CA         : /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    web
  Zone:       universe
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user

Connection Info:
  Socket Dir: /var/run/postgresql-web-universe-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User debezium_user does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: debezium_user
Storing password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: coordinator/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-02-05 07:12:25 UTC] USER=www-data EUID=0 PID=122288 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_web_universe_main_dev_db already exists
[2026-02-05 07:12:26 UTC] USER=www-data EUID=0 PID=122330 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d fastorder_web_universe_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=81c80b1b-f9eb-44f5-925f-0455b7853b9d)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (web) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=web
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
🔍 DEBUG_CHECKPOINT_03: Found service-specific run.sh for: web
[DEBUG] Tracking substep start: steps/01-install/steps/web (RUN_UUID=81c80b1b-f9eb-44f5-925f-0455b7853b9d)
[INFO] 🔸 Service: web (service-specific)
🔍 DEBUG_CHECKPOINT_04: Executing service-specific: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Web Service Schema Provisioning
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Service:     web
[INFO]   Zone:        universe
[INFO]   Branch:      main
[INFO]   Environment: dev
[INFO]   Identifier:  coordinator
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] 📦 Running: 01-init-schema.sh...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing config schema
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Identifier:  coordinator
  Database:    fastorder_web_universe_main_dev_db
  Host:        db-web-universe-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking if config schema exists...
[INFO] Creating config schema and tables...
CREATE EXTENSION
CREATE SCHEMA
CREATE TABLE
COMMENT
CREATE TABLE
COMMENT
CREATE TABLE
COMMENT
CREATE INDEX
CREATE INDEX
CREATE INDEX
INSERT 0 1
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
NOTICE:  trigger "trg_public_defaults_version" for relation "config.public_defaults" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
NOTICE:  trigger "trg_feature_flags_version" for relation "config.feature_flags" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
NOTICE:  trigger "trg_public_defaults_set_updated_at" for relation "config.public_defaults" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
NOTICE:  trigger "trg_feature_flags_set_updated_at" for relation "config.feature_flags" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
[OK] Config schema and tables created
[INFO] Seeding initial guest services data...
INSERT 0 9
INSERT 0 1
[OK] Initial data seeded
[INFO] Verifying config schema...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Config Schema Verification
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Guest services: 9
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] Config schema initialization complete
[OK] ✓ 01-init-schema.sh completed
[INFO] ═══════════════════════════════════════════════════════════════
[OK] Web service schema provisioning complete
[INFO] ═══════════════════════════════════════════════════════════════

✓ ✅ Coordinator setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up 1 worker(s) (Citus data nodes)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up worker: worker-01
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-02-05 07:12:30 UTC] USER=unknown EUID=33 PID=122594 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 07:12:30 UTC] USER=unknown EUID=33 PID=122601 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 07:12:30 UTC] USER=unknown EUID=33 PID=122608 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-02-05 07:12:30 UTC] USER=unknown EUID=33 PID=122615 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 07:12:30 UTC] USER=unknown EUID=33 PID=122625 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-02-05 07:12:30 UTC] USER=unknown EUID=33 PID=122648 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for web in universe-dev...
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=81c80b1b-f9eb-44f5-925f-0455b7853b9d JOB_UUID=ddd09ec2-cb6e-4c2d-83a4-2fe9cc94a9e7

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=81c80b1b-f9eb-44f5-925f-0455b7853b9d)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.243
[INFO] Primary hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01...
[INFO]   db-web-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.243

[INFO]   ✅ db-web-universe-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.243    db-web-universe-main-dev-postgresql-worker-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=81c80b1b-f9eb-44f5-925f-0455b7853b9d)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-web-universe-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      web-universe-main-dev.fastorder.com
  Internal IP: 10.100.1.243 (from topology: db-worker-01-postgresql)
  Shared IP:   142.93.238.16 (legacy/public, also included in SANs)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 07:12:33 UTC] USER=www-data EUID=0 PID=122831 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01 and /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:12:33 UTC] USER=www-data EUID=0 PID=122843 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-02-05 07:12:33 UTC] USER=www-data EUID=0 PID=122855 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-122770
[2026-02-05 07:12:33 UTC] USER=www-data EUID=0 PID=122868 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-122770/ra_root.crt
[2026-02-05 07:12:33 UTC] USER=www-data EUID=0 PID=122877 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-122770/ra_root.key
[2026-02-05 07:12:33 UTC] USER=www-data EUID=0 PID=122886 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-122770/ra_root.crt
[2026-02-05 07:12:33 UTC] USER=www-data EUID=0 PID=122895 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-122770/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[2026-02-05 07:12:35 UTC] USER=www-data EUID=0 PID=122948 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-122770/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 07:12:35 UTC] USER=www-data EUID=0 PID=122957 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-122770/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 07:12:35 UTC] USER=www-data EUID=0 PID=122966 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-02-05 07:12:35 UTC] USER=www-data EUID=0 PID=122975 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-122770/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:35 UTC] USER=www-data EUID=0 PID=122985 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:35 UTC] USER=www-data EUID=0 PID=122994 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:35 UTC] USER=www-data EUID=0 PID=123003 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 07:12:35 UTC] USER=www-data EUID=0 PID=123014 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 07:12:35 UTC] USER=www-data EUID=0 PID=123025 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 07:12:36 UTC] USER=www-data EUID=0 PID=123052 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 07:12:36 UTC] USER=www-data EUID=0 PID=123061 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 07:12:36 UTC] USER=www-data EUID=0 PID=123070 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:12:36 UTC] USER=www-data EUID=0 PID=123079 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01, DNS:localhost, IP Address:10.100.1.243, IP Address:127.0.0.1, IP Address:142.93.238.16
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: web-universe-main-dev
Node:        worker-01
Primary CN:  db-web-universe-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-web-universe-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:12:36 UTC] USER=www-data EUID=0 PID=123146 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-02-05 07:12:36 UTC] USER=www-data EUID=0 PID=123155 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 07:12:36 UTC] USER=www-data EUID=0 PID=123164 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-02-05 07:12:36 UTC] USER=www-data EUID=0 PID=123173 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 07:12:36 UTC] USER=www-data EUID=0 PID=123183 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123216 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123230 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123240 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123249 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123258 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123267 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123276 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123285 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123294 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123303 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123312 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123321 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123330 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123339 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123348 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123357 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123367 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123376 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123402 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123411 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:37 UTC] USER=www-data EUID=0 PID=123420 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123429 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123438 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123447 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123456 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123465 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123474 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123483 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123493 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123502 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123512 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123522 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123531 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123540 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123550 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123560 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123569 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123578 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123588 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123598 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123607 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123616 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123625 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123635 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123651 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123660 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123669 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123678 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123687 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123696 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123705 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123716 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123740 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123775 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123815 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123844 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123911 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:38 UTC] USER=www-data EUID=0 PID=123922 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=123934 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=123944 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=123953 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=123962 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=123971 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=123980 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=123990 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=123999 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=124008 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=124017 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=124028 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=124039 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=124050 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=124059 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=124068 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=124086 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=124096 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=124105 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=124116 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=124126 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:12:39 UTC] USER=www-data EUID=0 PID=124139 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: postgres
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:12:40 UTC] USER=www-data EUID=0 PID=124190 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-02-05 07:12:40 UTC] USER=www-data EUID=0 PID=124199 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 07:12:40 UTC] USER=www-data EUID=0 PID=124208 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-02-05 07:12:40 UTC] USER=www-data EUID=0 PID=124217 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 07:12:40 UTC] USER=www-data EUID=0 PID=124226 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124271 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124283 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124292 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124309 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124322 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124331 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124340 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124349 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124358 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124367 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124376 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124385 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124394 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124404 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124413 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124422 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124433 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124454 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124472 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124486 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:41 UTC] USER=www-data EUID=0 PID=124497 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124540 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124551 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124567 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124576 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124585 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124595 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124604 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124614 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124624 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124633 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124648 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124657 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124667 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124683 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124693 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124702 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124713 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124722 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124731 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124740 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124749 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124758 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124767 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124779 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124788 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124805 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124818 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:42 UTC] USER=www-data EUID=0 PID=124827 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=124836 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=124845 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=124854 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=124872 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=124882 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=124891 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=124901 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=124910 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=124919 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=124929 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=124940 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=124956 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=124972 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=124985 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=124994 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=125003 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=125027 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=125036 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=125045 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=125054 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=125063 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=125085 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=125094 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:12:43 UTC] USER=www-data EUID=0 PID=125103 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 07:12:44 UTC] USER=www-data EUID=0 PID=125112 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
   ✅ Symlinked client-key.pem
[2026-02-05 07:12:44 UTC] USER=www-data EUID=0 PID=125132 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:12:44 UTC] USER=www-data EUID=0 PID=125141 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:12:44 UTC] USER=www-data EUID=0 PID=125150 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:12:44 UTC] USER=www-data EUID=0 PID=125159 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: postgres
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=81c80b1b-f9eb-44f5-925f-0455b7853b9d)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-postgresql environment: db-web-universe-main-dev-postgresql-worker-01.fastorder.com (10.100.1.243)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.243
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: worker-01
[INFO] Data dir:   /data/postgresql/17/web-universe-main-dev/worker-01
[INFO] Port:       5432
[INFO] Hostname:   db-web-universe-main-dev-postgresql-worker-01
[2026-02-05 07:12:46 UTC] USER=www-data EUID=0 PID=125346 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:12:46 UTC] USER=www-data EUID=0 PID=125368 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:12:46 UTC] USER=www-data EUID=0 PID=125389 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:12:46 UTC] USER=www-data EUID=0 PID=125430 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-web-universe-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      web-universe-main-dev.fastorder.com
  Internal IP: 10.100.1.243 (from topology: db-worker-01-postgresql)
  Shared IP:   142.93.238.16 (legacy/public, also included in SANs)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 07:12:46 UTC] USER=www-data EUID=0 PID=125488 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01 and /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:12:46 UTC] USER=www-data EUID=0 PID=125498 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-02-05 07:12:46 UTC] USER=www-data EUID=0 PID=125508 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-125440
[2026-02-05 07:12:46 UTC] USER=www-data EUID=0 PID=125517 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-125440/ra_root.crt
[2026-02-05 07:12:46 UTC] USER=www-data EUID=0 PID=125529 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-125440/ra_root.key
[2026-02-05 07:12:46 UTC] USER=www-data EUID=0 PID=125540 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-125440/ra_root.crt
[2026-02-05 07:12:47 UTC] USER=www-data EUID=0 PID=125549 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-125440/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[2026-02-05 07:12:48 UTC] USER=www-data EUID=0 PID=125599 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-125440/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 07:12:48 UTC] USER=www-data EUID=0 PID=125609 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-125440/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 07:12:48 UTC] USER=www-data EUID=0 PID=125618 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-02-05 07:12:48 UTC] USER=www-data EUID=0 PID=125627 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-125440/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:48 UTC] USER=www-data EUID=0 PID=125636 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:48 UTC] USER=www-data EUID=0 PID=125652 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:12:48 UTC] USER=www-data EUID=0 PID=125661 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 07:12:48 UTC] USER=www-data EUID=0 PID=125672 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 07:12:48 UTC] USER=www-data EUID=0 PID=125681 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 07:12:48 UTC] USER=www-data EUID=0 PID=125690 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 07:12:48 UTC] USER=www-data EUID=0 PID=125699 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 07:12:48 UTC] USER=www-data EUID=0 PID=125708 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:12:48 UTC] USER=www-data EUID=0 PID=125717 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01, DNS:localhost, IP Address:10.100.1.243, IP Address:127.0.0.1, IP Address:142.93.238.16
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: web-universe-main-dev
Node:        worker-01
Primary CN:  db-web-universe-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-web-universe-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-02-05 07:12:48 UTC] USER=www-data EUID=0 PID=125746 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 07:12:48 UTC] USER=www-data EUID=0 PID=125755 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 07:12:48 UTC] USER=www-data EUID=0 PID=125764 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-02-05 07:12:49 UTC] USER=www-data EUID=0 PID=125785 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 07:12:49 UTC] USER=www-data EUID=0 PID=125819 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-web-universe-main-dev-worker-01
[2026-02-05 07:12:49 UTC] USER=www-data EUID=0 PID=125853 ACTION=fsop ARGS=rm -rf /var/run/postgresql-web-universe-main-dev-worker-01
[OK]   No conflicting Postgres left on port 5432
[OK]   Using postgres password from vault provider
[2026-02-05 07:12:51 UTC] USER=www-data EUID=0 PID=125944 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.dyKZDI
[2026-02-05 07:12:51 UTC] USER=www-data EUID=0 PID=125965 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.dyKZDI
[2026-02-05 07:12:51 UTC] USER=www-data EUID=0 PID=126002 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev
[2026-02-05 07:12:51 UTC] USER=www-data EUID=0 PID=126024 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev
[2026-02-05 07:12:51 UTC] USER=www-data EUID=0 PID=126047 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/web-universe-main-dev
[INFO] Initializing cluster in /data/postgresql/17/web-universe-main-dev/worker-01 (SCRAM; pwfile)
[WARN] Removing existing data directory: /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 07:12:51 UTC] USER=www-data EUID=0 PID=126077 ACTION=fsop ARGS=rm -rf /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 07:12:51 UTC] USER=www-data EUID=0 PID=126099 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 07:12:51 UTC] USER=www-data EUID=0 PID=126120 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 07:12:51 UTC] USER=www-data EUID=0 PID=126142 ACTION=fsop ARGS=chmod 700 /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 07:12:52 UTC] USER=www-data EUID=0 PID=126164 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-web-universe-main-dev-worker-01
[2026-02-05 07:12:52 UTC] USER=www-data EUID=0 PID=126215 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-web-universe-main-dev-worker-01
[2026-02-05 07:12:52 UTC] USER=www-data EUID=0 PID=126224 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/web-universe-main-dev/worker-01 --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.dyKZDI
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /data/postgresql/17/web-universe-main-dev/worker-01 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/web-universe-main-dev/worker-01 -l logfile start

[OK]   initdb complete
[2026-02-05 07:12:53 UTC] USER=www-data EUID=0 PID=126319 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.dyKZDI
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-02-05 07:12:53 UTC] USER=www-data EUID=0 PID=126374 ACTION=fsop ARGS=cp /tmp/tmp.f1e3G1MxAX /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
[2026-02-05 07:12:53 UTC] USER=www-data EUID=0 PID=126395 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
[2026-02-05 07:12:53 UTC] USER=www-data EUID=0 PID=126416 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 07:12:53 UTC] USER=www-data EUID=0 PID=126444 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.gG9ZLM /etc/systemd/system/postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 07:12:54 UTC] USER=www-data EUID=0 PID=126470 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@web-universe-main-dev-worker-01.service
[OK]   systemd unit written
[2026-02-05 07:12:54 UTC] USER=www-data EUID=0 PID=126514 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 07:12:54 UTC] USER=www-data EUID=0 PID=126537 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-02-05 07:12:56 UTC] USER=www-data EUID=0 PID=126700 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-worker-01.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-02-05 07:12:56 UTC] USER=www-data EUID=0 PID=126763 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-worker-01.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Provisioning application database and Debezium role (if not exists)...
[INFO] Checking if database fastorder_web_universe_main_dev_db exists...
[INFO] DB check result: exit_code=0, output='[2026-02-05 07:12:58 UTC] USER=www-data EUID=0 PID=126957 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_web_universe_main_dev_db''
[INFO] Creating database fastorder_web_universe_main_dev_db...
[2026-02-05 07:12:58 UTC] USER=www-data EUID=0 PID=126996 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_web_universe_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[OK]   Database fastorder_web_universe_main_dev_db created
[INFO] Checking if role debezium_user exists...
[INFO] Role check result: exit_code=0, output='[2026-02-05 07:12:58 UTC] USER=www-data EUID=0 PID=127032 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[INFO] Creating role debezium_user...
[2026-02-05 07:12:58 UTC] USER=www-data EUID=0 PID=127068 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'tMRZjm0T+iXJrDuQPBboyoeY';
CREATE ROLE
[OK]   Role debezium_user created
[2026-02-05 07:12:58 UTC] USER=www-data EUID=0 PID=127094 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_web_universe_main_dev_db" TO debezium_user;
GRANT
[OK]   Application DB (fastorder_web_universe_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[INFO] Applying connection and memory optimizations...
[INFO] Current settings: max_connections=100, work_mem=4MB
[INFO] Target settings (worker): max_connections=100, work_mem=8MB
[2026-02-05 07:12:59 UTC] USER=www-data EUID=0 PID=127174 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 100;
ALTER SYSTEM
[2026-02-05 07:12:59 UTC] USER=www-data EUID=0 PID=127208 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-02-05 07:12:59 UTC] USER=www-data EUID=0 PID=127232 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   Settings applied to postgresql.auto.conf
[2026-02-05 07:12:59 UTC] USER=www-data EUID=0 PID=127250 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/worker-01/standby.signal
[INFO] Service recently started (3s ago) - restarting to apply max_connections...
[INFO] Stopping service...
[2026-02-05 07:12:59 UTC] USER=www-data EUID=0 PID=127274 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-worker-01.service
[INFO] Waiting for port 5432 to be released...
[OK]   Port 5432 released
[INFO] Starting service...
[2026-02-05 07:13:03 UTC] USER=www-data EUID=0 PID=127381 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 07:13:09 UTC] USER=www-data EUID=0 PID=127616 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-worker-01.service
[OK]   ✅ Optimization complete: max_connections=100, work_mem=8MB
[OK]   Synchronous replication already configured (synchronous_commit: on)
[INFO] Setting postgres password via centralized script... for worker-01
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    web
  Zone:       universe
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01

Connection Info:
  Socket Dir: /var/run/postgresql-web-universe-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Initial setup: Using password from initdb
✓ PostgreSQL password already set during initdb
Storing password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/worker-01
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
[OK]   Password set and persisted
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.243
[INFO] Primary hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01...
[INFO]   db-web-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.243

[INFO]   ✅ db-web-universe-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.243    db-web-universe-main-dev-postgresql-worker-01.fastorder.com


[OK]   PostgreSQL 'web-universe-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key \
        host=db-web-universe-main-dev-postgresql-worker-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        web-universe-main-dev-postgresql-worker-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.243
[INFO]   Port:              5432
[INFO]   FQDN:              db-web-universe-main-dev-postgresql-worker-01
[INFO]   Status:            running
[INFO]   Environment:       web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 84e4185a-2ef1-49c1-8d2a-841d077f036b
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=81c80b1b-f9eb-44f5-925f-0455b7853b9d)
[INFO] 📦 03 role...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 07:13:19 UTC] USER=www-data EUID=0 PID=128229 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/worker-01/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    debezium_user
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   debezium_user
  Hostname:    db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:13:20 UTC] USER=www-data EUID=0 PID=128388 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-debezium_user
[2026-02-05 07:13:20 UTC] USER=www-data EUID=0 PID=128397 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-02-05 07:13:20 UTC] USER=www-data EUID=0 PID=128406 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-debezium_user/ra_root.key
[2026-02-05 07:13:20 UTC] USER=www-data EUID=0 PID=128417 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-02-05 07:13:20 UTC] USER=www-data EUID=0 PID=128426 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:13:20 UTC] USER=www-data EUID=0 PID=128444 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:13:20 UTC] USER=www-data EUID=0 PID=128454 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:13:20 UTC] USER=www-data EUID=0 PID=128470 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 07:13:20 UTC] USER=www-data EUID=0 PID=128482 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 07:13:20 UTC] USER=www-data EUID=0 PID=128496 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:20 UTC] USER=www-data EUID=0 PID=128509 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:13:20 UTC] USER=www-data EUID=0 PID=128518 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 07:13:20 UTC] USER=www-data EUID=0 PID=128533 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 07:13:20 UTC] USER=www-data EUID=0 PID=128543 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 07:13:20 UTC] USER=www-data EUID=0 PID=128552 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 07:13:20 UTC] USER=www-data EUID=0 PID=128561 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128570 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128579 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128588 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128597 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128606 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128615 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128624 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128652 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128661 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128670 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128679 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128700 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128709 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128718 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128727 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128736 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128745 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128754 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128772 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128787 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128796 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128807 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128817 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128826 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128835 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128844 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128853 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128862 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128871 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128880 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128889 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128899 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128909 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128918 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128927 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:21 UTC] USER=www-data EUID=0 PID=128936 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=128945 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=128954 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=128963 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=128972 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=128981 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=128993 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129020 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129030 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129040 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129050 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129063 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129076 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129085 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129097 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129106 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129115 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129124 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129133 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129142 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129151 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129160 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129170 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129180 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129189 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129198 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129207 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129216 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129225 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129234 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129243 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:13:22 UTC] USER=www-data EUID=0 PID=129252 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: debezium_user
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U debezium_user -d postgres

🔐 Generating replicator client certificate for worker-01...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129293 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129302 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129311 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129321 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129332 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129350 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129359 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129368 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129377 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129386 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129395 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129404 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129413 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129424 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129434 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129443 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129452 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129462 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129471 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129481 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129492 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129504 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129517 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129534 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129552 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:23 UTC] USER=www-data EUID=0 PID=129563 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129615 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129624 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129635 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129644 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129656 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129668 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129678 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129702 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129711 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129722 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129731 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129741 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129751 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129761 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129770 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129780 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129790 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129799 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129812 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129821 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129832 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129846 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129855 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129864 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129874 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129884 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129893 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129904 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129913 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129922 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:24 UTC] USER=www-data EUID=0 PID=129931 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=129940 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=129949 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=129958 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=129967 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=129976 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=129985 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=129995 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130007 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130016 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130025 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130034 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130043 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130054 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130063 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130072 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130081 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130090 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130101 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130110 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130122 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130145 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130165 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130179 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130191 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130200 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130209 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:13:25 UTC] USER=www-data EUID=0 PID=130220 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:13:26 UTC] USER=www-data EUID=0 PID=130235 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:13:26 UTC] USER=www-data EUID=0 PID=130244 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: replicator
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

✅ Replicator certificate generated for worker-01
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    web
  Zone:       universe
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd

Connection Info:
  Socket Dir: /var/run/postgresql-web-universe-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: fastorder_admin_gd
Storing password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: worker-01/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    fastorder_admin_gd
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   fastorder_admin_gd
  Hostname:    db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:13:34 UTC] USER=www-data EUID=0 PID=130881 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-fastorder_admin_gd
[2026-02-05 07:13:34 UTC] USER=www-data EUID=0 PID=130890 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-02-05 07:13:34 UTC] USER=www-data EUID=0 PID=130899 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
[2026-02-05 07:13:34 UTC] USER=www-data EUID=0 PID=130908 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-02-05 07:13:34 UTC] USER=www-data EUID=0 PID=130917 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:13:35 UTC] USER=www-data EUID=0 PID=130935 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:13:35 UTC] USER=www-data EUID=0 PID=130953 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 07:13:35 UTC] USER=www-data EUID=0 PID=130964 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 07:13:35 UTC] USER=www-data EUID=0 PID=130973 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:35 UTC] USER=www-data EUID=0 PID=130982 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:13:35 UTC] USER=www-data EUID=0 PID=130991 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:13:35 UTC] USER=www-data EUID=0 PID=131000 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 07:13:35 UTC] USER=www-data EUID=0 PID=131010 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 07:13:35 UTC] USER=www-data EUID=0 PID=131020 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 07:13:35 UTC] USER=www-data EUID=0 PID=131041 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 07:13:35 UTC] USER=www-data EUID=0 PID=131061 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:35 UTC] USER=www-data EUID=0 PID=131071 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 07:13:35 UTC] USER=www-data EUID=0 PID=131089 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:13:35 UTC] USER=www-data EUID=0 PID=131098 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 07:13:35 UTC] USER=www-data EUID=0 PID=131116 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131160 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131170 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131180 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131189 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131198 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131207 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131230 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131245 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131254 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131275 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131285 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131294 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131303 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131312 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131321 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131330 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131339 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131348 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131357 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131366 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131375 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131384 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:36 UTC] USER=www-data EUID=0 PID=131405 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131424 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131442 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131451 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131469 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131478 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131487 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131496 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131514 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131524 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131534 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131543 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131552 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131570 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131579 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131589 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131615 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131624 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131633 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131642 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131651 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131661 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131671 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131680 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131689 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
   ✅ Symlinked client-cert.pem
   ✅ Symlinked client-key.pem
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131716 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131725 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131734 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:13:37 UTC] USER=www-data EUID=0 PID=131743 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: fastorder_admin_gd
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-web-universe-main-dev-worker-01:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_web_universe_main_dev_db already exists, skipping creation
[2026-02-05 07:13:38 UTC] USER=www-data EUID=0 PID=131802 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-02-05 07:13:38 UTC] USER=www-data EUID=0 PID=131873 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-worker-01.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

=== Pre-flight Checks ===
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
✓ AWS Secrets Manager accessible

=== Retrieving Credentials from AWS ===
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials loaded for worker-01/fastorder_admin_gd: fastorder_admin_gd@db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_web_universe_main_dev_db
✓ Credentials retrieved: fastorder_admin_gd@db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_web_universe_main_dev_db
╔════════════════════════════════════════════╗
║  PostgreSQL Test Suite (AWS Secrets MGR)  ║
╚════════════════════════════════════════════╝

=== PostgreSQL Authentication Test ===
✗ PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-web-universe-main-dev-postgresql-worker-01.fastorder.com" (10.100.1.243), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 07:13:44 UTC] USER=www-data EUID=0 PID=132320 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/worker-01/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : web-universe-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_web_universe_main_dev_db
  SCHEMA      : web
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
  DNS → 10.100.1.243
  CA         : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    web
  Zone:       universe
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user

Connection Info:
  Socket Dir: /var/run/postgresql-web-universe-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User debezium_user does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: debezium_user
Storing password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: worker-01/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-02-05 07:13:52 UTC] USER=www-data EUID=0 PID=132749 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
ℹ️  Database fastorder_web_universe_main_dev_db already exists
[2026-02-05 07:13:52 UTC] USER=www-data EUID=0 PID=132784 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d fastorder_web_universe_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
🔐 Creating replicator role for worker-01...
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : web-universe-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SSL DIR     : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
  DNS → 10.100.1.243
  CA         : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Creating role: replicator with password
SET
CREATE ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/web/universe/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🎉 Done.
✅ Replicator role created for worker-01

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=81c80b1b-f9eb-44f5-925f-0455b7853b9d)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (web) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=web
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
🔍 DEBUG_CHECKPOINT_03: Found service-specific run.sh for: web
[DEBUG] Tracking substep start: steps/01-install/steps/web (RUN_UUID=81c80b1b-f9eb-44f5-925f-0455b7853b9d)
[INFO] 🔸 Service: web (service-specific)
🔍 DEBUG_CHECKPOINT_04: Executing service-specific: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Web Service Schema Provisioning
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Service:     web
[INFO]   Zone:        universe
[INFO]   Branch:      main
[INFO]   Environment: dev
[INFO]   Identifier:  coordinator
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] 📦 Running: 01-init-schema.sh...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing config schema
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Identifier:  coordinator
  Database:    fastorder_web_universe_main_dev_db
  Host:        db-web-universe-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking if config schema exists...
[INFO] Config schema already exists - checking tables...
[OK] Config schema with 3 tables already exists - skipping
[OK] ✓ 01-init-schema.sh completed
[INFO] ═══════════════════════════════════════════════════════════════
[OK] Web service schema provisioning complete
[INFO] ═══════════════════════════════════════════════════════════════

✓ ✅ Worker worker-01 setup completed

Skipping standbys (PG_WORKERS_STANDBY_NUM=0)
✓ ✅ PostgreSQL installation completed
[INFO] Discovering additional setup steps...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 02-pg-bouncer.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up PgBouncer connection pooling...
[2026-02-05 07:14:01 UTC] USER=www-data EUID=0 PID=133236 ACTION=fsop ARGS=rm -f /tmp/pgbouncer-ip.service /tmp/pgbouncer.service
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] Checking for existing PgBouncer application environment in topology …
[OK]   Using existing PgBouncer environment:
[INFO]   IP:     10.100.1.244
[INFO]   FQDN:   db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Domain: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO] Ensuring /etc/hosts entry for db-web-universe-main-dev-postgresql-bouncer.fastorder.com …
[OK]   /etc/hosts already contains entry for db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[WARN] IP 10.100.1.244 is assigned to multiple interfaces:
    inet 10.100.1.242/32 scope global lo
       valid_lft forever preferred_lft forever
    inet 10.100.1.244/32 scope global lo:pgbouncer
--
    inet 10.100.1.243/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.100.1.244/32 scope global eth0:244
[WARN] This may cause routing issues
[INFO] Final verification of /etc/hosts entry for db-web-universe-main-dev-postgresql-bouncer.fastorder.com …
[OK]   /etc/hosts correctly maps db-web-universe-main-dev-postgresql-bouncer.fastorder.com to 10.100.1.244
[OK]   PgBouncer IP 10.100.1.244 already correctly bound to lo:pgbouncer
[2026-02-05 07:14:02 UTC] USER=www-data EUID=0 PID=133359 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 07:14:03 UTC] USER=www-data EUID=0 PID=133462 ACTION=passthru ARGS=systemctl restart pgbouncer-ip@web-universe-main-dev.service
Job for pgbouncer-ip@web-universe-main-dev.service failed because the control process exited with error code.
See "systemctl status pgbouncer-ip@web-universe-main-dev.service" and "journalctl -xeu pgbouncer-ip@web-universe-main-dev.service" for details.
[2026-02-05 07:14:03 UTC] USER=www-data EUID=0 PID=133490 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer-ip@web-universe-main-dev.service
[WARN] pgbouncer-ip@web-universe-main-dev.service is not active
[WARN] Check status: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@web-universe-main-dev.service
[2026-02-05 07:14:03 UTC] USER=www-data EUID=0 PID=133530 ACTION=fsop ARGS=mkdir -p /etc/pgbouncer/web-universe-main-dev
[2026-02-05 07:14:03 UTC] USER=www-data EUID=0 PID=133539 ACTION=fsop ARGS=mkdir -p /run/pgbouncer/web-universe-main-dev
[2026-02-05 07:14:03 UTC] USER=www-data EUID=0 PID=133548 ACTION=fsop ARGS=mkdir -p /var/log/pgbouncer/web-universe-main-dev
[2026-02-05 07:14:03 UTC] USER=www-data EUID=0 PID=133557 ACTION=fsop ARGS=chmod 750 /etc/pgbouncer/web-universe-main-dev
[2026-02-05 07:14:03 UTC] USER=www-data EUID=0 PID=133566 ACTION=fsop ARGS=chmod 750 /run/pgbouncer/web-universe-main-dev
[2026-02-05 07:14:03 UTC] USER=www-data EUID=0 PID=133575 ACTION=fsop ARGS=chmod 750 /var/log/pgbouncer/web-universe-main-dev
[2026-02-05 07:14:03 UTC] USER=www-data EUID=0 PID=133584 ACTION=fsop ARGS=chown root:postgres /etc/pgbouncer/web-universe-main-dev
[2026-02-05 07:14:03 UTC] USER=www-data EUID=0 PID=133593 ACTION=fsop ARGS=chown postgres:postgres /run/pgbouncer/web-universe-main-dev
[2026-02-05 07:14:03 UTC] USER=www-data EUID=0 PID=133602 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbouncer/web-universe-main-dev
[INFO] Generating pgbouncer_admin client certificates...
[INFO] ⏳ This may take 30-60 seconds...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    pgbouncer_admin
Identifier:  pgbouncer
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        pgbouncer
  User (CN):   pgbouncer_admin
  Hostname:    db-web-universe-main-dev-postgresql-bouncer.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133639 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-pgbouncer-pgbouncer_admin
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133649 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133658 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133667 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133676 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = pgbouncer_admin
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133693 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133707 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133717 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133726 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133735 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133744 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133753 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133763 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133772 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133781 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133790 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133799 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 07:14:04 UTC] USER=www-data EUID=0 PID=133808 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=133817 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=133826 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=133835 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=133844 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=133854 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=133868 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=133877 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=133890 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=133926 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=133937 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=133946 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=133955 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=133964 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=133986 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=133995 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134004 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134013 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134022 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134031 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134041 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134053 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134062 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134071 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134080 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134090 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134099 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134108 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134119 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134142 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134151 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134165 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134176 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:05 UTC] USER=www-data EUID=0 PID=134188 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134197 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134206 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134215 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134224 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134233 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134242 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134251 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134260 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134269 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134278 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134287 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134299 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134309 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134318 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134327 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134336 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134345 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134354 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134363 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134372 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134381 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134390 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134399 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134408 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134418 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134428 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134437 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134446 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134455 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134464 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134473 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134484 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134493 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134502 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: pgbouncer_admin
Node: pgbouncer
FQDN: db-web-universe-main-dev-postgresql-bouncer.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-bouncer.fastorder.com -U pgbouncer_admin -d postgres

[OK]   mTLS client certificate present: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[INFO] Creating symlinks to canonical certificates in /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend...
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134518 ACTION=fsop ARGS=mkdir -p /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134527 ACTION=fsop ARGS=mkdir -p /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134536 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134545 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134554 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt
[INFO] Creating coordinator CA symlink for PostgreSQL server verification...
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134563 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] Verifying canonical certificate permissions...
[2026-02-05 07:14:06 UTC] USER=www-data EUID=0 PID=134572 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 07:14:07 UTC] USER=www-data EUID=0 PID=134581 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 07:14:07 UTC] USER=www-data EUID=0 PID=134590 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 07:14:07 UTC] USER=www-data EUID=0 PID=134599 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[OK]   Backend certificate symlinks created in /etc/ssl
[OK]   Coordinator CA symlink created for server verification
[OK]   Certificates already in canonical location - no symlinks needed
[2026-02-05 07:14:07 UTC] USER=www-data EUID=0 PID=134610 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
[2026-02-05 07:14:07 UTC] USER=www-data EUID=0 PID=134619 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
[2026-02-05 07:14:07 UTC] USER=www-data EUID=0 PID=134628 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 07:14:07 UTC] USER=www-data EUID=0 PID=134637 ACTION=fsop ARGS=test -r /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] PgBouncer will use PostgreSQL coordinator CA: /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[OK]   PostgreSQL coordinator at db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432 is reachable
[INFO] Dumping SCRAM secrets from coordinator for PgBouncer auth_file …
[2026-02-05 07:14:07 UTC] USER=www-data EUID=0 PID=134656 ACTION=fsop ARGS=cp /tmp/tmp.Jh7zGO2KXz /etc/pgbouncer/web-universe-main-dev/userlist.txt
[2026-02-05 07:14:07 UTC] USER=www-data EUID=0 PID=134665 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/userlist.txt
[2026-02-05 07:14:07 UTC] USER=www-data EUID=0 PID=134674 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/web-universe-main-dev/userlist.txt
[OK]   Auth file written: /etc/pgbouncer/web-universe-main-dev/userlist.txt
[INFO] Retrieved password from vault for pgbouncer_admin
[INFO] Ensuring PgBouncer admin role 'pgbouncer_admin' exists in Postgres (coordinator) …
[OK]   Role pgbouncer_admin created/updated successfully
[SECRETS] Setting credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/pgbouncer_admin
✓ [SECRETS] Credentials updated in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/pgbouncer_admin
[INFO] ✅ PgBouncer admin password stored in centralized secrets vault
[INFO] Re-fetching SCRAM secrets after role creation to ensure pgbouncer_admin is included …
[2026-02-05 07:14:13 UTC] USER=www-data EUID=0 PID=134951 ACTION=fsop ARGS=cp /tmp/tmp.LCT1ruHbyQ /etc/pgbouncer/web-universe-main-dev/userlist.txt
[2026-02-05 07:14:13 UTC] USER=www-data EUID=0 PID=134961 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/userlist.txt
[2026-02-05 07:14:13 UTC] USER=www-data EUID=0 PID=134970 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/web-universe-main-dev/userlist.txt
[OK]   Auth file updated with pgbouncer_admin SCRAM hash
[INFO] Auth file contains [2026-02-05 07:14:13 UTC] USER=www-data EUID=0 PID=135006 ACTION=passthru ARGS=bash -c wc -l < '/etc/pgbouncer/web-universe-main-dev/userlist.txt'
4 user(s)
[OK]   Admin 'pgbouncer_admin' password generated and saved
[INFO] Configuring PostgreSQL to prevent Citus metadata sync hangs...
ALTER ROLE
[OK]   Disabled Citus metadata sync for pgbouncer_admin
[INFO] Verifying application database fastorder_web_universe_main_dev_db exists...
[OK]   ✓ Database fastorder_web_universe_main_dev_db exists
[INFO] Granting permissions to pgbouncer_admin on fastorder_web_universe_main_dev_db...
GRANT
[OK]   ✓ Granted CONNECT on fastorder_web_universe_main_dev_db to pgbouncer_admin
GRANT
[OK]   ✓ Granted USAGE on schema public to pgbouncer_admin
GRANT
[OK]   ✓ Granted SELECT on all tables to pgbouncer_admin
ALTER DATABASE
[OK]   Set synchronous_commit=local for fastorder_web_universe_main_dev_db
[INFO] Ensuring pg_hba.conf entry for pgbouncer_admin …
[INFO] Adding pg_hba.conf entries for pgbouncer_admin with cert auth …
[2026-02-05 07:14:14 UTC] USER=unknown EUID=33 PID=135096 ACTION=-u ARGS=postgres bash
ERROR: Invalid or unauthorized action: -u
[OK]   pg_hba.conf updated and PostgreSQL configuration reloaded
[WARN] pg_hba.conf entry may not have loaded correctly
[INFO] Writing /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini …
[2026-02-05 07:14:15 UTC] USER=www-data EUID=0 PID=135165 ACTION=fsop ARGS=cp /tmp/tmp.sv0NUBR5vL /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
[2026-02-05 07:14:15 UTC] USER=www-data EUID=0 PID=135176 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
[2026-02-05 07:14:15 UTC] USER=www-data EUID=0 PID=135187 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
[2026-02-05 07:14:15 UTC] USER=www-data EUID=0 PID=135196 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbouncer/web-universe-main-dev /run/pgbouncer/web-universe-main-dev /var/log/pgbouncer/web-universe-main-dev
[2026-02-05 07:14:15 UTC] USER=www-data EUID=0 PID=135205 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/userlist.txt
[OK]   pgbouncer.ini ready
[INFO] Verifying TLS settings in pgbouncer.ini:
[2026-02-05 07:14:15 UTC] USER=www-data EUID=0 PID=135215 ACTION=fsop ARGS=grep -E (client_tls_sslmode|server_tls) /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
client_tls_sslmode = verify-full
server_tls_sslmode = verify-full
server_tls_ca_file = /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
server_tls_cert_file = /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
server_tls_key_file  = /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[INFO] Verifying PgBouncer server certificate files:
[2026-02-05 07:14:15 UTC] USER=www-data EUID=0 PID=135224 ACTION=fsop ARGS=test -r /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[OK]   Server cert readable by postgres: /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-02-05 07:14:15 UTC] USER=www-data EUID=0 PID=135233 ACTION=fsop ARGS=test -r /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[OK]   Server key readable by postgres: /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[INFO] Verifying coordinator CA certificate:
[2026-02-05 07:14:15 UTC] USER=www-data EUID=0 PID=135244 ACTION=fsop ARGS=test -r /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[OK]   Coordinator CA readable by postgres: /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] Preflight: stopping any conflicting PgBouncer on 6432 …
[2026-02-05 07:14:15 UTC] USER=www-data EUID=0 PID=135253 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer.service
[2026-02-05 07:14:15 UTC] USER=www-data EUID=0 PID=135262 ACTION=passthru ARGS=systemctl stop pgbouncer@web-universe-main-dev.service
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json?all=1": dial unix /var/run/docker.sock: connect: permission denied
[2026-02-05 07:14:17 UTC] USER=www-data EUID=0 PID=135350 ACTION=passthru ARGS=systemctl daemon-reload
[OK]   systemd unit installed: pgbouncer@web-universe-main-dev.service
[INFO] Running pre-flight IP conflict check for 10.100.1.244:6432 …
[WARN] IP conflict checker not found at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/check-ip-conflicts.sh
[WARN] Skipping pre-flight check - conflicts may occur
[INFO] Starting PgBouncer (web-universe-main-dev) …
[2026-02-05 07:14:19 UTC] USER=www-data EUID=0 PID=135466 ACTION=passthru ARGS=systemctl restart pgbouncer@web-universe-main-dev.service
[OK]   Service ACTIVE
[INFO] Verifying auth_file before probing …
[INFO] Auth file contains 4 user(s)
[WARN] Auth file does NOT contain pgbouncer_admin entry - authentication will fail
[INFO] Probing admin console via SSL (psql to database 'pgbouncer') …
[INFO] Retrieved password from vault for admin console probe
[OK]   Admin console reachable (SHOW POOLS OK)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Running Comprehensive PgBouncer Verification Tests
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Password extracted: WO0D2C0d7Z... (using postgres user certificates)

[INFO] Test 1/7: Admin Console - SHOW POOLS
 database  |   user    | cl_active | cl_waiting | cl_active_cancel_req | cl_waiting_cancel_req | sv_active | sv_active_cancel | sv_being_canceled | sv_idle | sv_used | sv_tested | sv_login | maxwait | maxwait_us | pool_mode | load_balance_hosts 
-----------+-----------+-----------+------------+----------------------+-----------------------+-----------+------------------+-------------------+---------+---------+-----------+----------+---------+------------+-----------+--------------------
 pgbouncer | pgbouncer |         1 |          0 |                    0 |                     0 |         0 |                0 |                 0 |       0 |       0 |         0 |        0 |       0 |          0 | statement | 
(1 row)

[OK]   ✓ SHOW POOLS: SUCCESS

[INFO] Test 2/7: Admin Console - SHOW VERSION
[OK]   ✓ SHOW VERSION: PgBouncer 1.24.1

[INFO] Test 3/7: Admin Console - SHOW STATS
 database  | total_server_assignment_count | total_xact_count | total_query_count | total_received | total_sent | total_xact_time | total_query_time | total_wait_time | total_client_parse_count | total_server_parse_count | total_bind_count | avg_server_assignment_count | avg_xact_count | avg_query_count | avg_recv | avg_sent | avg_xact_time | avg_query_time | avg_wait_time | avg_client_parse_count | avg_server_parse_count | avg_bind_count 
-----------+-------------------------------+------------------+-------------------+----------------+------------+-----------------+------------------+-----------------+--------------------------+--------------------------+------------------+-----------------------------+----------------+-----------------+----------+----------+---------------+----------------+---------------+------------------------+------------------------+----------------
 pgbouncer |                             0 |                4 |                 4 |              0 |          0 |               0 |                0 |               0 |                        0 |                        0 |                0 |                           0 |              0 |               0 |        0 |        0 |             0 |              0 |             0 |                      0 |                      0 |              0
(1 row)

[OK]   ✓ SHOW STATS: SUCCESS

[INFO] Test 4/7: Admin Console - SHOW DATABASES
                    name                     |                             host                              | port |              database              | force_user | pool_size | min_pool_size | reserve_pool_size | server_lifetime | pool_mode | load_balance_hosts | max_connections | current_connections | max_client_connections | current_client_connections | paused | disabled 
---------------------------------------------+---------------------------------------------------------------+------+------------------------------------+------------+-----------+---------------+-------------------+-----------------+-----------+--------------------+-----------------+---------------------+------------------------+----------------------------+--------+----------
 fastorder_web_universe_main_dev_db          | db-web-universe-main-dev-postgresql-coordinator.fastorder.com | 5432 | fastorder_web_universe_main_dev_db |            |       100 |             0 |                20 |            3600 |           |                    |               0 |                   0 |                      0 |                          0 |      0 |        0
 fastorder_web_universe_main_dev_db_worker_1 | pg-worker-01-web-universe-main-dev.fastorder.com              | 5432 | fastorder_web_universe_main_dev_db |            |       100 |             0 |                20 |            3600 |           |                    |               0 |                   0 |                      0 |                          0 |      0 |        0
 fastorder_web_universe_main_dev_db_worker_2 | pg-worker-01-standby-01-web-universe-main-dev.fastorder.com   | 5432 | fastorder_web_universe_main_dev_db |            |       100 |             0 |                20 |            3600 |           |                    |               0 |                   0 |                      0 |                          0 |      0 |        0
 fastorder_web_universe_main_dev_db_worker_3 | pg-worker-01-standby-02-web-universe-main-dev.fastorder.com   | 5432 | fastorder_web_universe_main_dev_db |            |       100 |             0 |                20 |            3600 |           |                    |               0 |                   0 |                      0 |                          0 |      0 |        0
 fastorder_web_universe_main_dev_db_worker_4 | pg-worker-01-standby-03-web-universe-main-dev.fastorder.com   | 5432 | fastorder_web_universe_main_dev_db |            |       100 |             0 |                20 |            3600 |           |                    |               0 |                   0 |                      0 |                          0 |      0 |        0
 pgbouncer                                   |                                                               | 6432 | pgbouncer                          | pgbouncer  |         2 |             0 |                 0 |            3600 | statement |                    |               0 |                   0 |                      0 |                          1 |      0 |        0
(6 rows)

[OK]   ✓ SHOW DATABASES: SUCCESS

[INFO] Test 5/7: Admin Console - SHOW CONFIG
[OK]   ✓ SHOW CONFIG: SUCCESS
[INFO]   Key settings:
[INFO]     client_tls_sslmode = verify-full|disable|yes
[INFO]     max_client_conn = 2048|100|yes
[INFO]     pool_mode = transaction|session|yes
[INFO]     server_tls_sslmode = verify-full|prefer|yes
psql   "host=db-web-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_web_universe_main_dev_db user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD    connect_timeout=5 sslmode=verify-full    sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt    sslcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt    sslkey=/etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key"   --no-psqlrc -Atc 'SELECT version();'

[INFO] Test 6/7: Application Database - SELECT version()
[WARN] ✗ Application database query: FAILED (timeout or connection issue)
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[INFO] Test 7/8: Application Database - Connection Details
[WARN] ✗ Connection details: FAILED (timeout or connection issue)
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[INFO] Test 8/8: End-to-End Application Routing - Pool Verification
[INFO]   Running actual queries through PgBouncer to verify routing and pooling...
[WARN] ✗ End-to-end routing verification: FAILED - All 3 queries failed
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[WARN]    Otherwise check if database fastorder_web_universe_main_dev_db exists and user pgbouncer_admin has permissions

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Verification Complete - Tests 1-5 PASSED (Admin console verified)
[WARN]   Tests 6-8 FAILED - Application database not accessible
[WARN]   This is expected if Citus is not set up yet
[WARN]   Run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[OK]   PgBouncer is up for web-universe-main-dev

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Connection Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Password stored in: AWS Secrets Manager (fastorder/db/web/ksa/main/dev/postgresqlweb/universe/main/dev/coordinator-pgbouncer_admin)
Current password: WO0D2C0d7ZbIdk65D10y9TaD

1. Admin Console (using IP address to avoid DNS/SSL issues):
   psql "host=10.100.1.244 port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD sslmode=verify-full sslrootcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

2. Admin Console (using hostname):
   psql "host=db-web-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD sslmode=verify-full sslrootcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

3. Application Database:
   psql "host=db-web-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_web_universe_main_dev_db sslkey=/etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD sslmode=verify-full sslrootcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

4. Using .pgpass file:
   echo "db-web-universe-main-dev-postgresql-bouncer.fastorder.com:6432:*:pgbouncer_admin:WO0D2C0d7ZbIdk65D10y9TaD" >> ~/.pgpass
   chmod 600 ~/.pgpass
   psql -h db-web-universe-main-dev-postgresql-bouncer.fastorder.com -p 6432 -U pgbouncer_admin -d fastorder_web_universe_main_dev_db

5. Retrieve password from vault:
   source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
   PGPASSWORD="$(get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password')" \
     psql -h 10.100.1.244 -p 6432 -U pgbouncer_admin -d pgbouncer -c "SHOW POOLS;"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Architecture
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  • Default db 'fastorder_web_universe_main_dev_db' → Citus coordinator (db-web-universe-main-dev-postgresql-coordinator.fastorder.com)
  • Worker access: 'fastorder_web_universe_main_dev_db_worker_1', 'fastorder_web_universe_main_dev_db_worker_2', … (if exist)
  • Client TLS: require (password auth) / verify-full (mTLS with certs)
  • Server TLS: verify-full (PgBouncer validates PostgreSQL certs)
  • Auth: SCRAM-SHA-256 via /etc/pgbouncer/web-universe-main-dev/userlist.txt
  • Pool mode: transaction (stateless connections)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Management
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Service Status:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer@web-universe-main-dev.service
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@web-universe-main-dev.service

Logs:
  command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@web-universe-main-dev.service -f
  /usr/local/bin/fastorder-provisioning-wrapper.sh tail -f /var/log/pgbouncer/web-universe-main-dev/pgbouncer.log

Reload Config:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@web-universe-main-dev.service

Restart:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@web-universe-main-dev.service

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Files
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Config:        /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
Auth file:     /etc/pgbouncer/web-universe-main-dev/userlist.txt
Server cert:   /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
Server key:    /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
CA cert:       /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt
PG CA:         /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
Logs:          /var/log/pgbouncer/web-universe-main-dev/pgbouncer.log

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Troubleshooting
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


If "SASL authentication failed":
  1. Check auth file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/web-universe-main-dev/userlist.txt
  2. Verify pgbouncer_admin is present with SCRAM hash
  3. Get password from vault:
     source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
     get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password'
  4. Reload PgBouncer: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@web-universe-main-dev.service

If "no pg_hba.conf entry":
  1. Check pg_hba.conf on coordinator
  2. Add rule: hostssl all pgbouncer_admin 10.100.1.244/32 cert clientcert=verify-full
  3. Reload PostgreSQL

To add users to PgBouncer:
  1. Create user in PostgreSQL with password
  2. Re-run SCRAM dump:
     psql "host=db-web-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 dbname=postgres user=postgres \
       sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt \
       sslcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt sslkey=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key" \
       -Atc "SELECT '\"' || rolname || '\" \"' || rolpassword || '\"' \
             FROM pg_authid WHERE rolpassword LIKE 'SCRAM-SHA-256%' \
             AND rolcanlogin ORDER BY rolname;" | command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop tee /etc/pgbouncer/web-universe-main-dev/userlist.txt
  3. Reload: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@web-universe-main-dev.service

[INFO] Registering PgBouncer node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        web-universe-main-dev-pgbouncer
[INFO]   Identifier Parent: postgresql
[INFO]   IP:                10.100.1.244
[INFO]   Port:              6432
[INFO]   FQDN:              db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: fdc70c5f-615d-432f-8161-a7acd56ea9ed
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[OK]   PgBouncer node registered to observability API
✓ ✅ PgBouncer setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 03-citus-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] CITUS DISTRIBUTED CLUSTER SETUP
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Phase 1: Installing Citus extension on workers...
[INFO] Phase 2: Setting up coordinator and registering workers...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 📦 PHASE 1: Installing Citus extension on 1 worker(s)...

[INFO] → Worker 1/1: Installing Citus on worker-01...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS CLUSTER SETUP
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔧 Setting up Citus Worker...
[INFO] Temporarily disabling synchronous replication for extension installation...
t
[INFO] Installing Citus extension on worker...
[OK]   Citus extension installed on worker
[INFO] Restoring synchronous replication settings...
t
[INFO] Worker Citus extension installed - registration will happen when coordinator setup runs

[OK]   Citus setup complete for worker-01
[INFO] ═══════════════════════════════════════════════════════════════════════════════
✓   ✅ Citus extension installed on worker-01

✓ ✅ Phase 1 Complete: All 1 workers have Citus extension installed

[INFO] 🔧 PHASE 2: Setting up Citus coordinator and registering workers...

[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS CLUSTER SETUP
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔧 Setting up Citus Coordinator...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] DIAGNOSTIC: Configuration Variables
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PG_WORKERS_NUM: 1
[INFO] ENV_ID: web-universe-main-dev
[INFO] DOMAIN: fastorder.com
[INFO] PORT: 5432
[INFO] SOCKET_DIR: /var/run/postgresql-web-universe-main-dev-coordinator
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Ensuring postgres client certificates exist for coordinator...
[OK]   Postgres client certificates already exist for coordinator
[INFO] Adding citus_cert_map to coordinator pg_ident.conf...
[OK]   pg_ident.conf updated for coordinator
[INFO] Installing Citus extension on coordinator...
[OK]   Citus extension installed on coordinator (postgres database)
[INFO] Installing Citus extension on application database: fastorder_web_universe_main_dev_db...
[OK]   Citus extension installed on application database: fastorder_web_universe_main_dev_db
[INFO] Configuring Citus SSL connection parameters...
[2026-02-05 07:14:49 UTC] USER=www-data EUID=0 PID=136732 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
[OK]   ✅ Citus SSL connection parameters configured: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[WARN] Node not identified as coordinator, initializing...
[INFO] Checking coordinator configuration...
[INFO] Persisting citus.local_hostname to postgresql.conf...
[2026-02-05 07:14:52 UTC] USER=www-data EUID=0 PID=136821 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/web-universe-main-dev/coordinator/postgresql.conf
[2026-02-05 07:14:52 UTC] USER=www-data EUID=0 PID=136843 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
[OK]   ✅ citus.local_hostname persisted to config and reloaded
[INFO] Configuring coordinator hostname in postgres database: db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432

[OK]   ✅ Coordinator hostname set to db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432 in postgres database
[INFO] Checking coordinator configuration in application database: fastorder_web_universe_main_dev_db...
[INFO] Configuring coordinator hostname in application database: db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432
[OK]   ✅ Coordinator hostname set to db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432 in application database
[INFO] Validating coordinator configuration before worker registration...
[OK]   ✅ Coordinator hostname validated: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[OK]   ✅ citus_tables view is accessible
[INFO] Checking coordinator self-registration...
[OK]   ✅ Coordinator is already self-registered
[INFO] Configuring coordinator shard placement policy...
[OK]   ✅ Coordinator already configured in postgres database (shouldhaveshards = false)
[OK]   ✅ Coordinator already configured in application database (shouldhaveshards = false)
[INFO] Registering 1 worker(s) to Citus cluster...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PRE-FLIGHT: Checking worker availability...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking worker worker-01...
[INFO]   FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[OK]   ✅ Worker worker-01 is reachable via SSL
[OK]   All workers are reachable - proceeding with registration

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Adding Citus worker: db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Adding citus_cert_map to worker-01 pg_ident.conf...
[OK]   pg_ident.conf updated for worker-01
[INFO] Configuring worker worker-01 HBA for coordinator (10.100.1.54) access...
[OK]   Worker worker-01 HBA configured for coordinator (10.100.1.54)
[INFO] Adding replication rules for 3 standby(s)...
[OK]   Replication rules added for worker-01
[INFO] Reloading worker worker-01 to apply HBA changes...
[2026-02-05 07:14:55 UTC] USER=www-data EUID=0 PID=137148 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-worker-01.service
[INFO] Configuring coordinator HBA for worker worker-01 (10.100.1.243) access...
[OK]   Coordinator HBA configured for worker worker-01 (10.100.1.243)
[INFO] Reloading coordinator to apply HBA changes...
[2026-02-05 07:14:55 UTC] USER=www-data EUID=0 PID=137181 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
[INFO] Ensuring postgres client certificates exist for worker-01...
[OK]   Postgres client certificates already exist for worker-01
[INFO] Configuring citus.node_conninfo on worker-01...
[2026-02-05 07:14:55 UTC] USER=www-data EUID=0 PID=137197 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-worker-01.service
[OK]   citus.node_conninfo configured on worker-01
[INFO] Temporarily relaxing sync-rep on worker worker-01...
t
[OK]   Worker worker-01 sync-rep relaxed (was: sync_commit=on)
[INFO] Ensuring Citus extension on worker databases...
CREATE EXTENSION
CREATE EXTENSION
[INFO] Running citus_add_node with 180s timeout...
2
[INFO] Restoring worker worker-01 sync-rep settings...
t
[OK]   Worker worker-01 sync-rep restored
[OK]   ✅ Worker db-web-universe-main-dev-postgresql-worker-01.fastorder.com successfully added to Citus cluster
[INFO]    Node ID: 2
[INFO]    Registered in: postgres, fastorder_web_universe_main_dev_db
[OK]   Worker worker-01 registration successful
[INFO] Configuring worker worker-01 shard placement policy...
[OK]   ✅ Worker worker-01 configured to hold shards in all databases


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] POST-REGISTRATION: Verifying cluster state...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Expected workers: 1
[INFO] Registered workers: 1
[OK]   ✅ All 1 workers successfully registered!

[INFO] Citus cluster configuration:
db-web-universe-main-dev-postgresql-coordinator.fastorder.com  5432  0  t  primary  f
db-web-universe-main-dev-postgresql-worker-01.fastorder.com    5432  1  t  primary  t

[INFO] Note: groupid=0 is the coordinator, groupid>0 are workers
[INFO]       shouldhaveshards: false=query router only, true=holds data shards

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] FINAL VALIDATION: Verifying configuration persistence...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:14:59 UTC] USER=www-data EUID=0 PID=137496 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/web-universe-main-dev/coordinator/postgresql.conf
[OK]   ✅ citus.local_hostname persisted in postgresql.conf
[OK]   ✅ All 1 worker(s) successfully registered and verified

[OK]   ✅ All validation checks passed
[OK]   Citus coordinator setup complete

[OK]   Citus setup complete for coordinator
[INFO] ═══════════════════════════════════════════════════════════════════════════════

✓ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ CITUS CLUSTER SETUP COMPLETED SUCCESSFULLY
✓    Coordinator: Ready and accepting connections
✓    Workers registered: 1
✓ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 05-backup-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up coordinator backup...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for web-universe-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-02-05 07:15:01 UTC] USER=www-data EUID=0 PID=137772 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 07:15:01 UTC] USER=www-data EUID=0 PID=137896 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-02-05 07:15:01 UTC] USER=www-data EUID=0 PID=137997 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-02-05 07:15:01 UTC] USER=www-data EUID=0 PID=138060 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-02-05 07:15:07 UTC] USER=www-data EUID=0 PID=139724 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-02-05 07:15:07 UTC] USER=www-data EUID=0 PID=139835 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-02-05 07:15:07 UTC] USER=www-data EUID=0 PID=139885 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 07:15:07 UTC] USER=www-data EUID=0 PID=139923 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-web-universe-main-dev
[2026-02-05 07:15:07 UTC] USER=www-data EUID=0 PID=139994 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-02-05 07:15:07 UTC] USER=www-data EUID=0 PID=140047 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[INFO] Ensuring correct ownership...
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-02-05 07:15:07 UTC] USER=www-data EUID=0 PID=140150 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-02-05 07:15:08 UTC] USER=www-data EUID=0 PID=140214 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-02-05 07:15:08 UTC] USER=www-data EUID=0 PID=140315 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-02-05 07:15:08 UTC] USER=www-data EUID=0 PID=140389 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/coordinator/PG_VERSION
[2026-02-05 07:15:08 UTC] USER=www-data EUID=0 PID=140425 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza web-universe-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-02-05 07:15:09 UTC] USER=www-data EUID=0 PID=140728 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-02-05 07:15:12 UTC] USER=www-data EUID=0 PID=141593 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 07:15:16 UTC] USER=www-data EUID=0 PID=143214 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-02-05 07:15:16 UTC] USER=www-data EUID=0 PID=143270 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator --log-level-console=info check
2026-02-05 07:15:16.613 P00   INFO: check command begin 2.56.0: --exec-id=143283-648a9afb --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 07:15:16.640 P00   INFO: check repo1 configuration (primary)
2026-02-05 07:15:16.657 P00  ERROR: [028]: backup and archive info files exist but do not match the database
                                    HINT: is this the correct stanza?
                                    HINT: did an error occur during stanza-upgrade?
2026-02-05 07:15:16.657 P00   INFO: check command end: aborted with exception [028]
[WARN] ⚠️  Stanza verification failed - this may be normal if WAL archiving hasn't started yet
[WARN]    The backup system is configured and will work once WAL segments are generated

[INFO] 8️⃣ Creating backup automation scripts...
[2026-02-05 07:15:16 UTC] USER=www-data EUID=0 PID=143338 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 07:15:16 UTC] USER=www-data EUID=0 PID=143357 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 07:15:16 UTC] USER=www-data EUID=0 PID=143398 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-02-05 07:15:17 UTC] USER=www-data EUID=0 PID=143474 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-web-universe-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-02-05 07:15:17 UTC] USER=www-data EUID=0 PID=143530 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 07:15:17 UTC] USER=www-data EUID=0 PID=143567 ACTION=fsop ARGS=sed -i s|__ENV_ID__|web-universe-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 07:15:17 UTC] USER=www-data EUID=0 PID=143598 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 07:15:17 UTC] USER=www-data EUID=0 PID=143607 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-02-05 07:15:17.395 P00   INFO: start command begin 2.56.0: --exec-id=143628-5ed8d05d --log-level-console=info --log-level-file=debug --stanza=web-universe-main-dev-coordinator
2026-02-05 07:15:17.396 P00   WARN: stop file does not exist for stanza web-universe-main-dev-coordinator
2026-02-05 07:15:17.396 P00   INFO: start command end: completed successfully (4ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-02-05 07:15:17.446 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=143639-8f7b44dc --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 07:15:17.447 P00   INFO: stanza-upgrade for stanza 'web-universe-main-dev-coordinator' on repo1
2026-02-05 07:15:17.460 P00   INFO: stanza-upgrade command end: completed successfully (17ms)
[INFO] This may take a few minutes depending on database size...
[2026-02-05 07:15:17 UTC] USER=www-data EUID=0 PID=143643 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260205-071517.log
[2026-02-05 07:15:17 UTC] USER=www-data EUID=0 PID=143663 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260205-071517.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-02-05 07:15:27 UTC] USER=www-data EUID=0 PID=144011 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-137581.log /var/log/pgbackrest/initial-backup-20260205-071517.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260205-071517.log
   2026-02-05 07:15:26.836 P00   INFO: new backup label = 20260205-071517F
   2026-02-05 07:15:26.941 P00   INFO: full backup size = 33.5MB, file total = 1441
   2026-02-05 07:15:26.941 P00   INFO: backup command end: completed successfully (9301ms)
   2026-02-05 07:15:26.941 P00   INFO: expire command begin 2.56.0: --exec-id=143680-aee5cdb7 --log-level-console=info --log-level-file=debug --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --repo1-retention-diff=4 --repo1-retention-full=4 --stanza=web-universe-main-dev-coordinator
   2026-02-05 07:15:26.944 P00   INFO: expire command end: completed successfully (3ms)

[INFO] Current backups:
stanza: web-universe-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000008

        full backup: 20260205-065853F
            timestamp start/stop: 2026-02-05 06:58:53+00 / 2026-02-05 06:58:56+00
            wal start/stop: 000000010000000000000005 / 000000010000000000000005
            database size: 33.5MB, database backup size: 33.5MB
            repo1: backup set size: 5.4MB, backup size: 5.4MB

    db (current)
        wal archive min/max (17): 000000010000000000000002/000000010000000000000003

        full backup: 20260205-071517F
            timestamp start/stop: 2026-02-05 07:15:17+00 / 2026-02-05 07:15:26+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 33.5MB, database backup size: 33.5MB
            repo1: backup set size: 5.4MB, backup size: 5.4MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         web-universe-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/web-universe-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up worker backups for 1 worker(s)...
[INFO] Setting up backup for: worker-01
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for web-universe-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-02-05 07:15:27 UTC] USER=www-data EUID=0 PID=144079 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 07:15:27 UTC] USER=www-data EUID=0 PID=144100 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-02-05 07:15:27 UTC] USER=www-data EUID=0 PID=144109 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-02-05 07:15:27 UTC] USER=www-data EUID=0 PID=144119 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-02-05 07:15:27 UTC] USER=www-data EUID=0 PID=144129 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-02-05 07:15:30 UTC] USER=www-data EUID=0 PID=144195 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-02-05 07:15:30 UTC] USER=www-data EUID=0 PID=144205 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-02-05 07:15:30 UTC] USER=www-data EUID=0 PID=144214 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-web-universe-main-dev
[2026-02-05 07:15:30 UTC] USER=www-data EUID=0 PID=144258 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-02-05 07:15:30 UTC] USER=www-data EUID=0 PID=144267 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-02-05 07:15:30 UTC] USER=www-data EUID=0 PID=144279 ACTION=fsop ARGS=find /data/postgresql/17/web-universe-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-02-05 07:15:30 UTC] USER=www-data EUID=0 PID=144291 ACTION=fsop ARGS=chown -R postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-02-05 07:15:30 UTC] USER=www-data EUID=0 PID=144302 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-02-05 07:15:31 UTC] USER=www-data EUID=0 PID=144312 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-02-05 07:15:31 UTC] USER=www-data EUID=0 PID=144341 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-02-05 07:15:31 UTC] USER=www-data EUID=0 PID=144358 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/coordinator/PG_VERSION
[2026-02-05 07:15:31 UTC] USER=www-data EUID=0 PID=144371 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza web-universe-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-02-05 07:15:32 UTC] USER=www-data EUID=0 PID=144484 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-02-05 07:15:34 UTC] USER=www-data EUID=0 PID=144533 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 07:15:38 UTC] USER=www-data EUID=0 PID=144641 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-02-05 07:15:38 UTC] USER=www-data EUID=0 PID=144665 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator --log-level-console=info check
2026-02-05 07:15:38.590 P00   INFO: check command begin 2.56.0: --exec-id=144673-edae187b --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 07:15:38.610 P00   INFO: check repo1 configuration (primary)
2026-02-05 07:15:38.665 P00   INFO: check repo1 archive for WAL (primary)
2026-02-05 07:15:38.966 P00   INFO: WAL segment 000000010000000000000005 successfully archived to '/var/lib/pgbackrest/backup/web-universe-main-dev/archive/web-universe-main-dev-coordinator/17-2/0000000100000000/000000010000000000000005-cf50d2096c61efca06bd7850adcc891edca9b248.lz4' on repo1
2026-02-05 07:15:38.966 P00   INFO: check command end: completed successfully (379ms)
[INFO] ✅ Stanza verification passed

[INFO] 8️⃣ Creating backup automation scripts...
[2026-02-05 07:15:39 UTC] USER=www-data EUID=0 PID=144695 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 07:15:39 UTC] USER=www-data EUID=0 PID=144705 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 07:15:39 UTC] USER=www-data EUID=0 PID=144723 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[2026-02-05 07:15:39 UTC] USER=www-data EUID=0 PID=144732 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-02-05 07:15:39 UTC] USER=www-data EUID=0 PID=144750 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-web-universe-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-02-05 07:15:39 UTC] USER=www-data EUID=0 PID=144768 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 07:15:39 UTC] USER=www-data EUID=0 PID=144777 ACTION=fsop ARGS=sed -i s|__ENV_ID__|web-universe-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 07:15:39 UTC] USER=www-data EUID=0 PID=144786 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/data/postgresql/17/web-universe-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 07:15:39 UTC] USER=www-data EUID=0 PID=144795 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 07:15:39 UTC] USER=www-data EUID=0 PID=144810 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-02-05 07:15:39.394 P00   INFO: start command begin 2.56.0: --exec-id=144836-9e9ecf9c --log-level-console=info --log-level-file=debug --stanza=web-universe-main-dev-coordinator
2026-02-05 07:15:39.394 P00   WARN: stop file does not exist for stanza web-universe-main-dev-coordinator
2026-02-05 07:15:39.394 P00   INFO: start command end: completed successfully (3ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-02-05 07:15:39.444 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=144848-12c3dcd2 --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 07:15:39.445 P00   INFO: stanza-upgrade for stanza 'web-universe-main-dev-coordinator' on repo1
2026-02-05 07:15:39.446 P00   INFO: stanza 'web-universe-main-dev-coordinator' on repo1 is already up to date
2026-02-05 07:15:39.446 P00   INFO: stanza-upgrade command end: completed successfully (6ms)
[INFO] This may take a few minutes depending on database size...
[2026-02-05 07:15:39 UTC] USER=www-data EUID=0 PID=144852 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260205-071539.log
[2026-02-05 07:15:39 UTC] USER=www-data EUID=0 PID=144861 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260205-071539.log
[2026-02-05 07:15:39 UTC] USER=www-data EUID=0 PID=144873 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260205-071539.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-02-05 07:15:41 UTC] USER=www-data EUID=0 PID=145082 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-144038.log /var/log/pgbackrest/initial-backup-20260205-071539.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260205-071539.log
   2026-02-05 07:15:41.418 P00   INFO: new backup label = 20260205-071539F
   2026-02-05 07:15:41.489 P00   INFO: full backup size = 33.5MB, file total = 1441
   2026-02-05 07:15:41.489 P00   INFO: backup command end: completed successfully (1928ms)
   2026-02-05 07:15:41.489 P00   INFO: expire command begin 2.56.0: --exec-id=144892-d342c750 --log-level-console=info --log-level-file=debug --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --repo1-retention-diff=4 --repo1-retention-full=4 --stanza=web-universe-main-dev-coordinator
   2026-02-05 07:15:41.490 P00   INFO: expire command end: completed successfully (1ms)

[INFO] Current backups:
stanza: web-universe-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000008

        full backup: 20260205-065853F
            timestamp start/stop: 2026-02-05 06:58:53+00 / 2026-02-05 06:58:56+00
            wal start/stop: 000000010000000000000005 / 000000010000000000000005
            database size: 33.5MB, database backup size: 33.5MB
            repo1: backup set size: 5.4MB, backup size: 5.4MB

    db (current)
        wal archive min/max (17): 000000010000000000000002/000000010000000000000005

        full backup: 20260205-071517F
            timestamp start/stop: 2026-02-05 07:15:17+00 / 2026-02-05 07:15:26+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 33.5MB, database backup size: 33.5MB
            repo1: backup set size: 5.4MB, backup size: 5.4MB

        full backup: 20260205-071539F
            timestamp start/stop: 2026-02-05 07:15:39+00 / 2026-02-05 07:15:41+00
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 33.5MB, database backup size: 33.5MB
            repo1: backup set size: 5.4MB, backup size: 5.4MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         web-universe-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/web-universe-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ Backup setup completed for coordinator and all workers

[INFO] Skipping 06-distribute-tables-canary.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 07-distribute-tables.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:15:43 UTC] USER=unknown EUID=33 PID=145198 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 07:15:43 UTC] USER=unknown EUID=33 PID=145207 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 07:15:43 UTC] USER=unknown EUID=33 PID=145214 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 07:15:43 UTC] USER=unknown EUID=33 PID=145222 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS TABLE DISTRIBUTION
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔐 Secure connection established
[INFO]    Host: db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432
[INFO]    Database: fastorder_web_universe_main_dev_db
[INFO]    SSL: verify-full (TLS 1.2+)
[INFO]    Timeouts: statement=120s, idle_tx=300s

[INFO] 🔍 Running preflight checks...
[INFO] Testing database connectivity...
[OK]   ✅ Database connection successful
[OK]   ✅ Connected to correct database: fastorder_web_universe_main_dev_db
[INFO] Checking Citus extension in database fastorder_web_universe_main_dev_db...
[OK]   Citus version: 13.2-1
[INFO] Checking worker registration...
[OK]   Registered workers: 1
[INFO] Worker nodes:
[INFO]                             nodename                           | nodeport | isactive | noderole 
[INFO]   -------------------------------------------------------------+----------+----------+----------
[INFO]    db-web-universe-main-dev-postgresql-worker-01.fastorder.com |     5432 | t        | primary
[INFO]   (1 row)
[INFO]   

[INFO] 📊 Starting table distribution...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Distributing: auth.login_account
[INFO] Description: User authentication table - distributed by region for tenant isolation
[INFO] Shard key: region_hint
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ⏭️  Table does not exist, skipping

[INFO] ═══════════════════════════════════════════════════════════════════════════════
[OK]   ✅ All tables distributed successfully!
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 📊 Citus Cluster Summary:

[INFO] Distributed tables:
[INFO]    table | type | shard_key | shards | size 
[INFO]   -------+------+-----------+--------+------
[INFO]   (0 rows)
[INFO]   

[INFO] Worker capacity:
[INFO]    worker | total_shards | total_size 
[INFO]   --------+--------------+------------
[INFO]   (0 rows)
[INFO]   

[OK]   Citus table distribution complete

[INFO] Skipping 08-distribute-tables-rollback.sh (rollback script - run manually only)
[INFO] Skipping 09-distribute-tables-test.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 10-setup-cdc.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] CDC PIPELINE SETUP (Debezium + Elasticsearch Sink)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Log file: /var/log/fastorder/cdc/10-setup-cdc-*.log

[INFO] Running CDC setup for identifier: coordinator
[2026-02-05 07:15:47] ==========================================
[2026-02-05 07:15:47] CDC SETUP SCRIPT STARTED
[2026-02-05 07:15:47] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260205_071547.log
[2026-02-05 07:15:47] ==========================================
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 07:15:47] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:15:47]   CDC Pipeline Setup (Debezium + ES Sink)
[2026-02-05 07:15:47] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:15:47]   Environment: web-universe-main-dev
[2026-02-05 07:15:47]   Identifier:  coordinator
[2026-02-05 07:15:47]   Service:     web
[2026-02-05 07:15:47] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:15:47] 📂 CDC_BASE_DIR exists: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc
[2026-02-05 07:15:47] Looking for service folder: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web
[2026-02-05 07:15:47] 
[2026-02-05 07:15:47] 📂 Found CDC configuration for service: web
[2026-02-05 07:15:47] Scanning for subservice directories in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web
[2026-02-05 07:15:47] Found subservice: config, checking for steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps
[2026-02-05 07:15:47] 
[2026-02-05 07:15:47] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:15:47]   Setting up CDC for: web/config
[2026-02-05 07:15:47] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 07:15:47] Found 3 step script(s) in /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps
[2026-02-05 07:15:47] 
[2026-02-05 07:15:47] 🔧 Running: 01-setup-config-cdc.sh
[2026-02-05 07:15:47]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps/01-setup-config-cdc.sh
[2026-02-05 07:15:47]    Executing directly (script is executable)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Config CDC Setup - Automatic Role Detection
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Service:     web
[INFO]   Zone:        universe
[INFO]   Branch:      main
[INFO]   Environment: dev
[INFO]   Identifier:  coordinator
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service 'web' requires config.* schema
[INFO] CDC Role for web in zone universe: master
[INFO] ════════════════════════════════════════════════════════════
[INFO]   CONTROL PLANE MASTER (zone=universe)
[INFO]   Setting up Debezium CDC Publisher
[INFO] ════════════════════════════════════════════════════════════
[INFO] Executing Debezium config setup...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Debezium Config CDC Master Setup
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Environment:      web-universe-main-dev
[INFO]   PostgreSQL Host:  db-web-universe-main-dev-postgresql.fastorder.com
[INFO]   Database:         fastorder_web_universe_main_dev_db
[INFO]   Connector:        debezium-web-universe-main-dev-config
[INFO]   Kafka Connect:    eventbus-web-universe-main-dev-kafka-connect.fastorder.com:8083
[INFO]   SSL Cert Dir:     /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator (on Kafka Connect host)
[INFO]   SSL Key File:     /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der
[INFO]   Topic Prefix:     cdc.web_universe_main_dev
[INFO]   Replication Slot: dbz_web_universe_main_dev_config
[INFO]   Tables:           config.public_defaults,config.feature_flags,config.config_version
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Step 1: Creating PostgreSQL publication for config tables...
[INFO] Creating publication: cdc_pub_web_config
CREATE PUBLICATION
[INFO] Setting REPLICA IDENTITY FULL for config tables...
ALTER TABLE
ALTER TABLE
ALTER TABLE
[INFO] Step 2: Creating replication slot...
[INFO] Creating replication slot: dbz_web_universe_main_dev_config
 (dbz_web_universe_main_dev_config,0/700E1D8)

[INFO] Step 3: Registering Debezium connector with Kafka Connect...
[INFO] Creating new connector: debezium-web-universe-main-dev-config
[INFO] Sending connector configuration to Kafka Connect...
[ OK ] Debezium connector registered successfully
[INFO] Step 4: Verifying connector status...
[INFO]   Connector State: RUNNING
[INFO]   Task State:      RUNNING
[ OK ] Debezium connector is running
[INFO] 
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Debezium Config CDC Master Setup Complete
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Connector:        debezium-web-universe-main-dev-config
[INFO]   Topic Prefix:     cdc.web_universe_main_dev
[INFO]   Publication:      cdc_pub_web_config
[INFO]   Replication Slot: dbz_web_universe_main_dev_config
[INFO] 
[INFO]   Topics Created:
[INFO]     - cdc.web_universe_main_dev.config.public_defaults
[INFO]     - cdc.web_universe_main_dev.config.feature_flags
[INFO]     - cdc.web_universe_main_dev.config.config_version
[INFO] 
[INFO]   Data Planes (replicas) should subscribe to:
[INFO]     Topic Prefix: cdc.web_universe_main_dev
[INFO] ═══════════════════════════════════════════════════════════════
[ OK ] Debezium config CDC master setup complete
[ OK ] Debezium config CDC master setup complete
[INFO] No topology.json found at /opt/fastorder/state/web-universe-main-dev/topology.json - skipping merge
[INFO] 
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Config CDC Setup Complete
[INFO] ═══════════════════════════════════════════════════════════════
═══════════════════════════════════════════════════════════════
  Service Capabilities: web-universe-main-dev
═══════════════════════════════════════════════════════════════

  Service:          web
  Zone:             universe
  Branch:           main
  Environment:      dev

  Config Schema:    ✅ YES
  Redis Cache:      ✅ YES
  CDC Role:         master

  CDC Master Configuration:
    Debezium:       debezium-web-universe-main-dev-config
    Topic Prefix:   cdc.web_universe_main_dev
    Repl Slot:      dbz_web_universe_main_dev_config
    Tables:         config.public_defaults,config.feature_flags,config.config_version

  Required Schemas: config tenant dashboard environment resource service item company communication ai
═══════════════════════════════════════════════════════════════
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] 
[INFO] Log file: /var/log/fastorder/cdc/config-cdc-20260205_071547.log
[ OK ] Config CDC setup finished successfully
[2026-02-05 07:15:54] ✅ Completed: 01-setup-config-cdc.sh
[2026-02-05 07:15:54] 
[2026-02-05 07:15:54] 🔧 Running: 02-setup-debezium-config.sh
[2026-02-05 07:15:54]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps/02-setup-debezium-config.sh
[2026-02-05 07:15:54]    Executing directly (script is executable)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Debezium Config CDC Master Setup
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Environment:      web-universe-main-dev
[INFO]   PostgreSQL Host:  db-web-universe-main-dev-postgresql.fastorder.com
[INFO]   Database:         fastorder_web_universe_main_dev_db
[INFO]   Connector:        debezium-web-universe-main-dev-config
[INFO]   Kafka Connect:    eventbus-web-universe-main-dev-kafka-connect.fastorder.com:8083
[INFO]   SSL Cert Dir:     /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator (on Kafka Connect host)
[INFO]   SSL Key File:     /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der
[INFO]   Topic Prefix:     cdc.web_universe_main_dev
[INFO]   Replication Slot: dbz_web_universe_main_dev_config
[INFO]   Tables:           config.public_defaults,config.feature_flags,config.config_version
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Step 1: Creating PostgreSQL publication for config tables...
[INFO] Publication cdc_pub_web_config already exists
[INFO] Setting REPLICA IDENTITY FULL for config tables...
ALTER TABLE
ALTER TABLE
ALTER TABLE
[INFO] Step 2: Creating replication slot...
[INFO] Replication slot dbz_web_universe_main_dev_config already exists
[INFO] Step 3: Registering Debezium connector with Kafka Connect...
[INFO] Connector debezium-web-universe-main-dev-config already exists - updating configuration
[INFO] Sending connector configuration to Kafka Connect...
[ OK ] Debezium connector registered successfully
[INFO] Step 4: Verifying connector status...
[INFO]   Connector State: RUNNING
[INFO]   Task State:      RUNNING
[ OK ] Debezium connector is running
[INFO] 
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Debezium Config CDC Master Setup Complete
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Connector:        debezium-web-universe-main-dev-config
[INFO]   Topic Prefix:     cdc.web_universe_main_dev
[INFO]   Publication:      cdc_pub_web_config
[INFO]   Replication Slot: dbz_web_universe_main_dev_config
[INFO] 
[INFO]   Topics Created:
[INFO]     - cdc.web_universe_main_dev.config.public_defaults
[INFO]     - cdc.web_universe_main_dev.config.feature_flags
[INFO]     - cdc.web_universe_main_dev.config.config_version
[INFO] 
[INFO]   Data Planes (replicas) should subscribe to:
[INFO]     Topic Prefix: cdc.web_universe_main_dev
[INFO] ═══════════════════════════════════════════════════════════════
[ OK ] Debezium config CDC master setup complete
[2026-02-05 07:16:00] ✅ Completed: 02-setup-debezium-config.sh
[2026-02-05 07:16:00] 
[2026-02-05 07:16:00] 🔧 Running: 03-setup-kafka-consumer.sh
[2026-02-05 07:16:00]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps/03-setup-kafka-consumer.sh
[2026-02-05 07:16:00]    Executing directly (script is executable)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[ERROR] This script should only run on Data Planes (zone!=universe)
[ERROR] Current zone: universe
[ERROR] Use 02-setup-debezium-config.sh for Control Plane
[2026-02-05 07:16:01] ❌ FAILED: 03-setup-kafka-consumer.sh (exit code: 1)
[2026-02-05 07:16:01] ❌ CRITICAL: This is a required step for CDC pipeline. Aborting.

[ERROR] ❌ Database infrastructure (postgresql) setup failed with exit code: 1
📥 Download Full Logs
9
06-finalizing local
⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars) 
Loading logs...
9
Total Steps
0
Succeeded
1
Failed
0
Running
8
Pending
5 minutes
Total Steps Time
← Back to Dashboard 🔍 View Environment