Provisioning Job Status

📝 Execution Steps (9)

0/9 completed 1 failed

0% (0/9 steps)

00-preflight-checks local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

00-terraform-provision local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

01-prepare-environment local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

02-iam local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

02-observability-cell local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

03-search local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

04-eventbus local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

05-db local

❌ FAILED

⏰ Started: 2026-02-05 05:43:21

🏁 Finished: 2026-02-05 05:49:52

⏱️ Duration: 6 minutes

📄 View Logs (391718 chars)

[0;34m[INFO][0m Using database engine from DB_ENGINE environment variable: postgresql
[0;34m[INFO][0m Cleaning up any existing locks...

[0;32m[1mStarting database engine: postgresql[0m
[1;33m═══════════════════════════════════════════════[0m

[0;34m[INFO][0m Using environment from web interface: web-universe-main-dev
[0;32m[2026-02-05 05:43:21][0m Using web-provided environment: web-universe-main-dev
[0;32m[2026-02-05 05:43:21][0m Service: web, Zone: universe, Branch: main, Env: dev
[0;32m✓[0m Environment initialized successfully (mode: general)
[0;34m[INFO][0m Checking observability cell readiness: obs-web-universe-main-dev
[1;32m[OK][0m   Observability cell endpoints registered for web-universe-main-dev
[0;34m[INFO][0m Observability cell verified for web-universe-main-dev
[0;34m[INFO][0m Monitoring will be configured after PostgreSQL deployment (step 10-monitoring-setup.sh)
[0;34m[INFO][0m Citus mode ENABLED
[0;34m[INFO][0m → Coordinator + 1 worker(s) + 0 standby node(s) per worker
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up coordinator (Citus control plane)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-02-05 05:43:22 UTC] USER=unknown EUID=33 PID=3924257 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 05:43:22 UTC] USER=unknown EUID=33 PID=3924264 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 05:43:22 UTC] USER=unknown EUID=33 PID=3924271 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-02-05 05:43:22 UTC] USER=unknown EUID=33 PID=3924278 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 05:43:22 UTC] USER=unknown EUID=33 PID=3924285 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-02-05 05:43:22 UTC] USER=unknown EUID=33 PID=3924292 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for web in universe-dev...
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=4b33b2ad-c806-4665-b381-eaab322ba3f8 JOB_UUID=582390f3-f4ed-4901-a786-456f7c46742a

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=4b33b2ad-c806-4665-b381-eaab322ba3f8)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.119
[0;34m[INFO][0m Primary hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com

[0;34m[INFO][0m Adding /etc/hosts entries for coordinator...
[0;34m[INFO][0m   1. db-web-universe-main-dev-postgresql.fastorder.com → 10.100.1.119 (primary/short)
[0;34m[INFO][0m   2. db-web-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.119 (compatibility)

[0;34m[INFO][0m   ✅ db-web-universe-main-dev-postgresql.fastorder.com already exists with correct IP
[0;34m[INFO][0m   ✅ db-web-universe-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;32m✅[0m   ✅ Network & DNS configuration complete
[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
  10.100.1.119    db-web-universe-main-dev-postgresql.fastorder.com
  10.100.1.119    db-web-universe-main-dev-postgresql-coordinator.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=4b33b2ad-c806-4665-b381-eaab322ba3f8)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-web-universe-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      web-universe-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 05:43:26 UTC] USER=www-data EUID=0 PID=3924469 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator and /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:43:26 UTC] USER=www-data EUID=0 PID=3924478 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-02-05 05:43:26 UTC] USER=www-data EUID=0 PID=3924488 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-3924413
[2026-02-05 05:43:26 UTC] USER=www-data EUID=0 PID=3924509 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-3924413/ra_root.key
[2026-02-05 05:43:26 UTC] USER=www-data EUID=0 PID=3924531 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3924413/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[2026-02-05 05:43:27 UTC] USER=www-data EUID=0 PID=3924599 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3924413/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 05:43:27 UTC] USER=www-data EUID=0 PID=3924617 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-02-05 05:43:28 UTC] USER=www-data EUID=0 PID=3924626 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3924413/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:28 UTC] USER=www-data EUID=0 PID=3924635 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:28 UTC] USER=www-data EUID=0 PID=3924644 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:28 UTC] USER=www-data EUID=0 PID=3924653 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 05:43:28 UTC] USER=www-data EUID=0 PID=3924664 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 05:43:28 UTC] USER=www-data EUID=0 PID=3924675 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 05:43:28 UTC] USER=www-data EUID=0 PID=3924685 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 05:43:28 UTC] USER=www-data EUID=0 PID=3924703 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-web-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:db-web-universe-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-web-universe-main-dev-postgresql.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: web-universe-main-dev
Node:        coordinator
Primary CN:  db-web-universe-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-web-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 05:43:29 UTC] USER=www-data EUID=0 PID=3924778 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-02-05 05:43:29 UTC] USER=www-data EUID=0 PID=3924787 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 05:43:29 UTC] USER=www-data EUID=0 PID=3924811 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-02-05 05:43:29 UTC] USER=www-data EUID=0 PID=3924820 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 05:43:29 UTC] USER=www-data EUID=0 PID=3924837 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:43:29 UTC] USER=www-data EUID=0 PID=3924873 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:43:29 UTC] USER=www-data EUID=0 PID=3924883 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:43:29 UTC] USER=www-data EUID=0 PID=3924892 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 05:43:29 UTC] USER=www-data EUID=0 PID=3924901 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 05:43:29 UTC] USER=www-data EUID=0 PID=3924915 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:29 UTC] USER=www-data EUID=0 PID=3924926 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 05:43:30 UTC] USER=www-data EUID=0 PID=3924939 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 05:43:30 UTC] USER=www-data EUID=0 PID=3924951 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 05:43:30 UTC] USER=www-data EUID=0 PID=3924962 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 05:43:30 UTC] USER=www-data EUID=0 PID=3924980 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 05:43:30 UTC] USER=www-data EUID=0 PID=3924992 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:30 UTC] USER=www-data EUID=0 PID=3925014 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 05:43:30 UTC] USER=www-data EUID=0 PID=3925024 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 05:43:30 UTC] USER=www-data EUID=0 PID=3925033 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 05:43:30 UTC] USER=www-data EUID=0 PID=3925043 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 05:43:30 UTC] USER=www-data EUID=0 PID=3925052 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:31 UTC] USER=www-data EUID=0 PID=3925075 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:31 UTC] USER=www-data EUID=0 PID=3925123 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:31 UTC] USER=www-data EUID=0 PID=3925141 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:43:31 UTC] USER=www-data EUID=0 PID=3925153 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:43:31 UTC] USER=www-data EUID=0 PID=3925181 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 05:43:31 UTC] USER=www-data EUID=0 PID=3925196 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 05:43:31 UTC] USER=www-data EUID=0 PID=3925205 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:31 UTC] USER=www-data EUID=0 PID=3925216 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 05:43:31 UTC] USER=www-data EUID=0 PID=3925225 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 05:43:31 UTC] USER=www-data EUID=0 PID=3925234 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 05:43:31 UTC] USER=www-data EUID=0 PID=3925243 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 05:43:31 UTC] USER=www-data EUID=0 PID=3925253 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:32 UTC] USER=www-data EUID=0 PID=3925263 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:32 UTC] USER=www-data EUID=0 PID=3925273 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:43:32 UTC] USER=www-data EUID=0 PID=3925285 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:43:32 UTC] USER=www-data EUID=0 PID=3925294 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:32 UTC] USER=www-data EUID=0 PID=3925305 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:32 UTC] USER=www-data EUID=0 PID=3925314 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 05:43:32 UTC] USER=www-data EUID=0 PID=3925333 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:32 UTC] USER=www-data EUID=0 PID=3925362 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 05:43:32 UTC] USER=www-data EUID=0 PID=3925383 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:32 UTC] USER=www-data EUID=0 PID=3925394 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:32 UTC] USER=www-data EUID=0 PID=3925403 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:43:33 UTC] USER=www-data EUID=0 PID=3925415 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:43:33 UTC] USER=www-data EUID=0 PID=3925426 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:33 UTC] USER=www-data EUID=0 PID=3925437 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:33 UTC] USER=www-data EUID=0 PID=3925459 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 05:43:33 UTC] USER=www-data EUID=0 PID=3925473 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:33 UTC] USER=www-data EUID=0 PID=3925483 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 05:43:33 UTC] USER=www-data EUID=0 PID=3925494 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 05:43:33 UTC] USER=www-data EUID=0 PID=3925503 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 05:43:33 UTC] USER=www-data EUID=0 PID=3925524 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:33 UTC] USER=www-data EUID=0 PID=3925534 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:33 UTC] USER=www-data EUID=0 PID=3925545 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:43:33 UTC] USER=www-data EUID=0 PID=3925554 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:43:34 UTC] USER=www-data EUID=0 PID=3925563 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:34 UTC] USER=www-data EUID=0 PID=3925581 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 05:43:34 UTC] USER=www-data EUID=0 PID=3925590 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 05:43:34 UTC] USER=www-data EUID=0 PID=3925628 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 05:43:34 UTC] USER=www-data EUID=0 PID=3925661 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:43:35 UTC] USER=www-data EUID=0 PID=3925672 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:43:35 UTC] USER=www-data EUID=0 PID=3925681 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:43:35 UTC] USER=www-data EUID=0 PID=3925690 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 05:43:35 UTC] USER=www-data EUID=0 PID=3925699 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 05:43:35 UTC] USER=www-data EUID=0 PID=3925708 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 05:43:35 UTC] USER=www-data EUID=0 PID=3925717 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:43:35 UTC] USER=www-data EUID=0 PID=3925727 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:43:35 UTC] USER=www-data EUID=0 PID=3925737 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:43:35 UTC] USER=www-data EUID=0 PID=3925748 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: postgres
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 05:43:36 UTC] USER=www-data EUID=0 PID=3925846 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-02-05 05:43:36 UTC] USER=www-data EUID=0 PID=3925856 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 05:43:36 UTC] USER=www-data EUID=0 PID=3925867 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-02-05 05:43:36 UTC] USER=www-data EUID=0 PID=3925877 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 05:43:36 UTC] USER=www-data EUID=0 PID=3925887 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3925922 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3925931 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3925940 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3925958 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3925967 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3925986 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3925995 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3926004 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3926014 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3926026 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3926035 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3926046 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3926055 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3926065 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3926079 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3926093 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3926102 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 05:43:37 UTC] USER=www-data EUID=0 PID=3926111 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:38 UTC] USER=www-data EUID=0 PID=3926120 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:38 UTC] USER=www-data EUID=0 PID=3926159 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:43:38 UTC] USER=www-data EUID=0 PID=3926168 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:43:38 UTC] USER=www-data EUID=0 PID=3926177 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:38 UTC] USER=www-data EUID=0 PID=3926186 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:38 UTC] USER=www-data EUID=0 PID=3926195 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 05:43:38 UTC] USER=www-data EUID=0 PID=3926205 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 05:43:38 UTC] USER=www-data EUID=0 PID=3926214 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:38 UTC] USER=www-data EUID=0 PID=3926233 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 05:43:38 UTC] USER=www-data EUID=0 PID=3926243 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 05:43:38 UTC] USER=www-data EUID=0 PID=3926263 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:38 UTC] USER=www-data EUID=0 PID=3926274 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:38 UTC] USER=www-data EUID=0 PID=3926292 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:43:38 UTC] USER=www-data EUID=0 PID=3926301 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:38 UTC] USER=www-data EUID=0 PID=3926310 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:38 UTC] USER=www-data EUID=0 PID=3926319 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926338 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926347 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926356 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926384 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926397 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926409 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926418 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926427 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926436 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926445 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926456 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926466 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926477 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926488 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926497 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926507 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926516 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 05:43:39 UTC] USER=www-data EUID=0 PID=3926526 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926536 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926545 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926555 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926564 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926584 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926594 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926603 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926615 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926632 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926643 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926652 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926662 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926677 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926687 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
   ✅ Symlinked ca.pem
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926705 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
   ✅ Symlinked client-key.pem
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926725 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926734 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926743 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:43:40 UTC] USER=www-data EUID=0 PID=3926752 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: postgres
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=4b33b2ad-c806-4665-b381-eaab322ba3f8)
[INFO] 📦 02 setup pg instance...
[0;34m[DEADLOCK-PREVENTION][0m Deadlock prevention library loaded
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-coordinator-postgresql environment: db-web-universe-main-dev-postgresql-coordinator.fastorder.com (10.100.1.119)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.119
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m Data dir:   /data/postgresql/17/web-universe-main-dev/coordinator
[0;34m[INFO][0m Port:       5432
[0;34m[INFO][0m Hostname:   db-web-universe-main-dev-postgresql-coordinator
[2026-02-05 05:43:42 UTC] USER=www-data EUID=0 PID=3926924 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:43:42 UTC] USER=www-data EUID=0 PID=3926946 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:43:43 UTC] USER=www-data EUID=0 PID=3926967 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:43:43 UTC] USER=www-data EUID=0 PID=3926988 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[1;33m[WARN][0m Server certificate not found at /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[0;34m[INFO][0m Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-web-universe-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      web-universe-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 05:43:43 UTC] USER=www-data EUID=0 PID=3927030 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator and /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:43:43 UTC] USER=www-data EUID=0 PID=3927039 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-02-05 05:43:43 UTC] USER=www-data EUID=0 PID=3927049 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-3926995
[2026-02-05 05:43:43 UTC] USER=www-data EUID=0 PID=3927067 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-3926995/ra_root.key
[2026-02-05 05:43:43 UTC] USER=www-data EUID=0 PID=3927076 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3926995/ra_root.crt
[2026-02-05 05:43:43 UTC] USER=www-data EUID=0 PID=3927086 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3926995/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[2026-02-05 05:43:47 UTC] USER=www-data EUID=0 PID=3927222 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3926995/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 05:43:47 UTC] USER=www-data EUID=0 PID=3927232 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3926995/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 05:43:47 UTC] USER=www-data EUID=0 PID=3927253 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-02-05 05:43:47 UTC] USER=www-data EUID=0 PID=3927269 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3926995/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:47 UTC] USER=www-data EUID=0 PID=3927285 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:43:47 UTC] USER=www-data EUID=0 PID=3927313 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 05:43:47 UTC] USER=www-data EUID=0 PID=3927325 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 05:43:47 UTC] USER=www-data EUID=0 PID=3927334 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 05:43:47 UTC] USER=www-data EUID=0 PID=3927344 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 05:43:47 UTC] USER=www-data EUID=0 PID=3927353 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 05:43:47 UTC] USER=www-data EUID=0 PID=3927363 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:43:47 UTC] USER=www-data EUID=0 PID=3927381 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-web-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:db-web-universe-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-web-universe-main-dev-postgresql.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: web-universe-main-dev
Node:        coordinator
Primary CN:  db-web-universe-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-web-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-02-05 05:43:48 UTC] USER=www-data EUID=0 PID=3927410 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 05:43:48 UTC] USER=www-data EUID=0 PID=3927419 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 05:43:48 UTC] USER=www-data EUID=0 PID=3927428 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[0;32m[OK][0m   mTLS certificates OK (server cert + client certs verified) and keys secured
[0;34m[INFO][0m Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-02-05 05:43:48 UTC] USER=www-data EUID=0 PID=3927449 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 05:43:48 UTC] USER=www-data EUID=0 PID=3927474 ACTION=passthru ARGS=systemctl stop postgresql
[1;33m[WARN][0m Cleaning stale socket directory /var/run/postgresql-web-universe-main-dev-coordinator
[2026-02-05 05:43:48 UTC] USER=www-data EUID=0 PID=3927510 ACTION=fsop ARGS=rm -rf /var/run/postgresql-web-universe-main-dev-coordinator
[0;32m[OK][0m   No conflicting Postgres left on port 5432
[0;32m[OK][0m   Using postgres password from vault provider
[2026-02-05 05:43:50 UTC] USER=www-data EUID=0 PID=3927622 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.lhVGki
[2026-02-05 05:43:50 UTC] USER=www-data EUID=0 PID=3927644 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.lhVGki
[2026-02-05 05:43:50 UTC] USER=www-data EUID=0 PID=3927671 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev
[2026-02-05 05:43:50 UTC] USER=www-data EUID=0 PID=3927700 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev
[2026-02-05 05:43:51 UTC] USER=www-data EUID=0 PID=3927725 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/web-universe-main-dev
[0;34m[INFO][0m Initializing cluster in /data/postgresql/17/web-universe-main-dev/coordinator (SCRAM; pwfile)
[1;33m[WARN][0m Removing existing data directory: /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 05:43:51 UTC] USER=www-data EUID=0 PID=3927746 ACTION=fsop ARGS=rm -rf /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 05:43:51 UTC] USER=www-data EUID=0 PID=3927788 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 05:43:51 UTC] USER=www-data EUID=0 PID=3927810 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 05:43:51 UTC] USER=www-data EUID=0 PID=3927871 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-web-universe-main-dev-coordinator
[2026-02-05 05:43:51 UTC] USER=www-data EUID=0 PID=3927904 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-web-universe-main-dev-coordinator
[2026-02-05 05:43:51 UTC] USER=www-data EUID=0 PID=3927925 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-web-universe-main-dev-coordinator
[2026-02-05 05:43:52 UTC] USER=www-data EUID=0 PID=3927940 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/web-universe-main-dev/coordinator --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.lhVGki
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /data/postgresql/17/web-universe-main-dev/coordinator ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/web-universe-main-dev/coordinator -l logfile start

[0;32m[OK][0m   initdb complete
[2026-02-05 05:43:53 UTC] USER=www-data EUID=0 PID=3928018 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.lhVGki
[0;34m[INFO][0m Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[0;32m[OK][0m   postgresql.conf updated successfully
[0;34m[INFO][0m Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-02-05 05:43:53 UTC] USER=www-data EUID=0 PID=3928065 ACTION=fsop ARGS=cp /tmp/tmp.z0YHOYrZ3c /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
[2026-02-05 05:43:53 UTC] USER=www-data EUID=0 PID=3928086 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
[2026-02-05 05:43:53 UTC] USER=www-data EUID=0 PID=3928107 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
[0;32m[OK][0m   pg_hba.conf updated
[0;34m[INFO][0m Creating systemd unit: /etc/systemd/system/postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 05:43:53 UTC] USER=www-data EUID=0 PID=3928132 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.MVRBXi /etc/systemd/system/postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 05:43:53 UTC] USER=www-data EUID=0 PID=3928156 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m   systemd unit written
[2026-02-05 05:43:53 UTC] USER=www-data EUID=0 PID=3928177 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 05:43:53 UTC] USER=www-data EUID=0 PID=3928198 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 05:43:53 UTC] USER=www-data EUID=0 PID=3928220 ACTION=passthru ARGS=systemctl daemon-reload
[0;34m[INFO][0m Starting PostgreSQL instance...
[2026-02-05 05:43:55 UTC] USER=www-data EUID=0 PID=3928361 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[0;34m[INFO][0m Waiting for ACTIVE (systemd)…
[2026-02-05 05:43:55 UTC] USER=www-data EUID=0 PID=3928404 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m   Service ACTIVE
[0;34m[INFO][0m Waiting for port 5432 bind…
[0;32m[OK][0m   Port bound
[0;34m[INFO][0m Waiting pg_isready (socket)…
[0;32m[OK][0m   Readiness via socket OK
[0;34m[INFO][0m Waiting pg_isready (TCP db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432)…
[0;32m[OK][0m   Startup sequence complete
[0;34m[INFO][0m Validating core security GUCs (via local socket)…
[0;32m[OK][0m   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[0;34m[INFO][0m Provisioning application database and Debezium role (if not exists)...
[0;34m[INFO][0m Checking if database fastorder_web_universe_main_dev_db exists...
[0;34m[INFO][0m DB check result: exit_code=0, output='[2026-02-05 05:43:56 UTC] USER=www-data EUID=0 PID=3928591 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_web_universe_main_dev_db''
[0;34m[INFO][0m Creating database fastorder_web_universe_main_dev_db...
[2026-02-05 05:43:57 UTC] USER=www-data EUID=0 PID=3928616 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_web_universe_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[0;32m[OK][0m   Database fastorder_web_universe_main_dev_db created
[0;34m[INFO][0m Checking if role debezium_user exists...
[0;34m[INFO][0m Role check result: exit_code=0, output='[2026-02-05 05:43:57 UTC] USER=www-data EUID=0 PID=3928646 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[0;34m[INFO][0m Creating role debezium_user...
[2026-02-05 05:43:57 UTC] USER=www-data EUID=0 PID=3928673 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'jBpQ4eUzky1PQHc6IhS50g8q';
CREATE ROLE
[0;32m[OK][0m   Role debezium_user created
[2026-02-05 05:43:57 UTC] USER=www-data EUID=0 PID=3928699 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_web_universe_main_dev_db" TO debezium_user;
GRANT
[0;32m[OK][0m   Application DB (fastorder_web_universe_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[0;34m[INFO][0m Applying connection and memory optimizations...
[0;34m[INFO][0m Current settings: max_connections=100, work_mem=4MB
[0;34m[INFO][0m Target settings (coordinator): max_connections=150, work_mem=8MB
[2026-02-05 05:43:58 UTC] USER=www-data EUID=0 PID=3928849 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 150;
ALTER SYSTEM
[2026-02-05 05:43:58 UTC] USER=www-data EUID=0 PID=3928878 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
[2026-02-05 05:43:58 UTC] USER=www-data EUID=0 PID=3928907 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[0;32m[OK][0m   Settings applied to postgresql.auto.conf
[2026-02-05 05:43:58 UTC] USER=www-data EUID=0 PID=3928922 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/coordinator/standby.signal
[0;34m[INFO][0m Service recently started (3s ago) - restarting to apply max_connections...
[0;34m[INFO][0m Stopping service...
[2026-02-05 05:43:58 UTC] USER=www-data EUID=0 PID=3928944 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[0;34m[INFO][0m Waiting for port 5432 to be released...
[0;32m[OK][0m   Port 5432 released
[0;34m[INFO][0m Starting service...
[2026-02-05 05:44:02 UTC] USER=www-data EUID=0 PID=3929156 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 05:44:08 UTC] USER=www-data EUID=0 PID=3929405 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m   ✅ Optimization complete: max_connections=150, work_mem=8MB
[0;34m[INFO][0m Setting postgres password via centralized script... for coordinator
[0;34m[INFO][0m Temporarily disabling synchronous_commit on coordinator for password setting...
[0;32m[OK][0m   Disabled synchronous_commit (was: on)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    web[0m
[0;34m  Zone:       universe[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: coordinator[0m

[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-web-universe-main-dev-coordinator[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mInitial setup: Using password from initdb[0m
[0;32m✓ PostgreSQL password already set during initdb[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/coordinator
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/coordinator[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials coordinator[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
[0;34m[INFO][0m Restoring synchronous_commit on coordinator...
[0;32m[OK][0m   Restored synchronous_commit to: on
[0;32m[OK][0m   Password set and persisted
[0;34m[INFO][0m Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.119
[0;34m[INFO][0m Primary hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com

[0;34m[INFO][0m Adding /etc/hosts entries for coordinator...
[0;34m[INFO][0m   1. db-web-universe-main-dev-postgresql.fastorder.com → 10.100.1.119 (primary/short)
[0;34m[INFO][0m   2. db-web-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.119 (compatibility)

[0;34m[INFO][0m   ✅ db-web-universe-main-dev-postgresql.fastorder.com already exists with correct IP
[0;34m[INFO][0m   ✅ db-web-universe-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;32m✅[0m   ✅ Network & DNS configuration complete
[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
  10.100.1.119    db-web-universe-main-dev-postgresql.fastorder.com
  10.100.1.119    db-web-universe-main-dev-postgresql-coordinator.fastorder.com


[0;32m[OK][0m   PostgreSQL 'web-universe-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key \
        host=db-web-universe-main-dev-postgresql-coordinator port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[0;34m[INFO][0m Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        web-universe-main-dev-postgresql-coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.119
[INFO]   Port:              5432
[INFO]   FQDN:              db-web-universe-main-dev-postgresql-coordinator
[INFO]   Status:            running
[INFO]   Environment:       web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 90feffd7-89fb-4afb-a63f-cc975d7e928c
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[0;32m[OK][0m   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=4b33b2ad-c806-4665-b381-eaab322ba3f8)
[INFO] 📦 03 role...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 05:44:22 UTC] USER=www-data EUID=0 PID=3930367 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/coordinator/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    debezium_user
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   debezium_user
  Hostname:    db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 05:44:23 UTC] USER=www-data EUID=0 PID=3930550 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-debezium_user
[2026-02-05 05:44:24 UTC] USER=www-data EUID=0 PID=3930560 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-02-05 05:44:24 UTC] USER=www-data EUID=0 PID=3930582 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-02-05 05:44:24 UTC] USER=www-data EUID=0 PID=3930597 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:44:24 UTC] USER=www-data EUID=0 PID=3930615 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:44:24 UTC] USER=www-data EUID=0 PID=3930624 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:44:24 UTC] USER=www-data EUID=0 PID=3930633 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 05:44:24 UTC] USER=www-data EUID=0 PID=3930642 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 05:44:25 UTC] USER=www-data EUID=0 PID=3930651 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:44:25 UTC] USER=www-data EUID=0 PID=3930660 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 05:44:25 UTC] USER=www-data EUID=0 PID=3930669 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 05:44:25 UTC] USER=www-data EUID=0 PID=3930678 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 05:44:25 UTC] USER=www-data EUID=0 PID=3930687 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 05:44:25 UTC] USER=www-data EUID=0 PID=3930699 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 05:44:25 UTC] USER=www-data EUID=0 PID=3930709 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:44:25 UTC] USER=www-data EUID=0 PID=3930718 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:44:25 UTC] USER=www-data EUID=0 PID=3930733 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 05:44:25 UTC] USER=www-data EUID=0 PID=3930743 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 05:44:25 UTC] USER=www-data EUID=0 PID=3930752 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 05:44:25 UTC] USER=www-data EUID=0 PID=3930761 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 05:44:25 UTC] USER=www-data EUID=0 PID=3930776 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:44:25 UTC] USER=www-data EUID=0 PID=3930785 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:25 UTC] USER=www-data EUID=0 PID=3930812 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:25 UTC] USER=www-data EUID=0 PID=3930837 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:44:25 UTC] USER=www-data EUID=0 PID=3930847 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:26 UTC] USER=www-data EUID=0 PID=3930873 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 05:44:26 UTC] USER=www-data EUID=0 PID=3930882 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 05:44:26 UTC] USER=www-data EUID=0 PID=3930909 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 05:44:26 UTC] USER=www-data EUID=0 PID=3930919 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 05:44:26 UTC] USER=www-data EUID=0 PID=3930930 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 05:44:26 UTC] USER=www-data EUID=0 PID=3930939 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 05:44:26 UTC] USER=www-data EUID=0 PID=3930953 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:26 UTC] USER=www-data EUID=0 PID=3930985 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:44:26 UTC] USER=www-data EUID=0 PID=3930994 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:44:26 UTC] USER=www-data EUID=0 PID=3931003 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:26 UTC] USER=www-data EUID=0 PID=3931012 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:26 UTC] USER=www-data EUID=0 PID=3931021 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 05:44:26 UTC] USER=www-data EUID=0 PID=3931045 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:44:26 UTC] USER=www-data EUID=0 PID=3931056 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931074 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931099 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931109 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931122 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931134 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931146 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931155 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931168 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931177 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931186 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931204 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931213 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931222 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931231 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931242 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931252 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931263 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931272 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931281 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931290 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931301 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 05:44:27 UTC] USER=www-data EUID=0 PID=3931311 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 05:44:28 UTC] USER=www-data EUID=0 PID=3931338 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 05:44:28 UTC] USER=www-data EUID=0 PID=3931347 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 05:44:28 UTC] USER=www-data EUID=0 PID=3931366 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 05:44:28 UTC] USER=www-data EUID=0 PID=3931378 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:44:28 UTC] USER=www-data EUID=0 PID=3931388 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:44:28 UTC] USER=www-data EUID=0 PID=3931406 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 05:44:28 UTC] USER=www-data EUID=0 PID=3931415 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 05:44:28 UTC] USER=www-data EUID=0 PID=3931424 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 05:44:28 UTC] USER=www-data EUID=0 PID=3931433 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:44:28 UTC] USER=www-data EUID=0 PID=3931443 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:44:28 UTC] USER=www-data EUID=0 PID=3931452 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:44:28 UTC] USER=www-data EUID=0 PID=3931461 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: debezium_user
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U debezium_user -d postgres

✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    web[0m
[0;34m  Zone:       universe[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: coordinator[0m

[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-web-universe-main-dev-coordinator[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mGenerating new secure password...[0m
[0;34mUser fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: fastorder_admin_gd[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials coordinator[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: coordinator/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-web-universe-main-dev-postgresql.fastorder.com
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    fastorder_admin_gd
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   fastorder_admin_gd
  Hostname:    db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 05:44:38 UTC] USER=www-data EUID=0 PID=3932072 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-fastorder_admin_gd
[2026-02-05 05:44:39 UTC] USER=www-data EUID=0 PID=3932081 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-02-05 05:44:39 UTC] USER=www-data EUID=0 PID=3932091 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
[2026-02-05 05:44:39 UTC] USER=www-data EUID=0 PID=3932105 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-02-05 05:44:39 UTC] USER=www-data EUID=0 PID=3932116 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:44:39 UTC] USER=www-data EUID=0 PID=3932137 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:44:39 UTC] USER=www-data EUID=0 PID=3932164 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 05:44:39 UTC] USER=www-data EUID=0 PID=3932174 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 05:44:39 UTC] USER=www-data EUID=0 PID=3932192 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 05:44:39 UTC] USER=www-data EUID=0 PID=3932201 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 05:44:39 UTC] USER=www-data EUID=0 PID=3932210 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 05:44:39 UTC] USER=www-data EUID=0 PID=3932219 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932228 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932237 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932248 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932257 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932269 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932280 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932292 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932311 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932320 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932329 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932338 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932364 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932375 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932384 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932398 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932411 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932420 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932429 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932439 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932448 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932459 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932473 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932488 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932498 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:40 UTC] USER=www-data EUID=0 PID=3932508 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932518 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932527 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932536 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932545 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932558 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932567 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932576 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932585 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932613 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932623 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932633 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932642 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932651 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932660 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932669 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932680 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932689 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932701 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932718 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932727 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932736 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932745 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932755 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932765 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932774 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932783 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932792 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932801 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932810 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932819 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932828 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932840 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 05:44:41 UTC] USER=www-data EUID=0 PID=3932849 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 05:44:42 UTC] USER=www-data EUID=0 PID=3932858 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 05:44:42 UTC] USER=www-data EUID=0 PID=3932867 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:44:42 UTC] USER=www-data EUID=0 PID=3932887 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:44:42 UTC] USER=www-data EUID=0 PID=3932896 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:44:42 UTC] USER=www-data EUID=0 PID=3932907 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 05:44:42 UTC] USER=www-data EUID=0 PID=3932925 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 05:44:42 UTC] USER=www-data EUID=0 PID=3932950 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 05:44:42 UTC] USER=www-data EUID=0 PID=3932965 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:44:42 UTC] USER=www-data EUID=0 PID=3932976 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:44:42 UTC] USER=www-data EUID=0 PID=3932985 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: fastorder_admin_gd
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-web-universe-main-dev-coordinator:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_web_universe_main_dev_db already exists, skipping creation
[2026-02-05 05:44:42 UTC] USER=www-data EUID=0 PID=3933129 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-02-05 05:44:42 UTC] USER=www-data EUID=0 PID=3933168 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

[0;34m=== Pre-flight Checks ===[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;32m✓[0m AWS Secrets Manager accessible

[0;34m=== Retrieving Credentials from AWS ===[0m
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials loaded for coordinator/fastorder_admin_gd: fastorder_admin_gd@db-web-universe-main-dev-postgresql.fastorder.com:5432/fastorder_web_universe_main_dev_db
[0;32m✓[0m Credentials retrieved: fastorder_admin_gd@db-web-universe-main-dev-postgresql.fastorder.com:5432/fastorder_web_universe_main_dev_db
[0;34m╔════════════════════════════════════════════╗[0m
[0;34m║  PostgreSQL Test Suite (AWS Secrets MGR)  ║[0m
[0;34m╚════════════════════════════════════════════╝[0m

[0;34m=== PostgreSQL Authentication Test ===[0m
[0;31m✗[0m PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-web-universe-main-dev-postgresql.fastorder.com" (10.100.1.119), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
── fast setup ─────────────────────────────────────────────
  NAME        : web-universe-main-dev
  IDENTIFIER  : coordinator
  PG HOST     : db-web-universe-main-dev-postgresql.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_web_universe_main_dev_db
  SCHEMA      : web
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
  DNS → 10.100.1.119
  CA         : /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    web[0m
[0;34m  Zone:       universe[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: coordinator[0m

[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-web-universe-main-dev-coordinator[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mGenerating new secure password...[0m
[0;34mUser debezium_user does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: debezium_user[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials coordinator[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: coordinator/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-02-05 05:44:56 UTC] USER=www-data EUID=0 PID=3934109 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_web_universe_main_dev_db already exists
[2026-02-05 05:44:57 UTC] USER=www-data EUID=0 PID=3934140 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d fastorder_web_universe_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=4b33b2ad-c806-4665-b381-eaab322ba3f8)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (web) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=web
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
🔍 DEBUG_CHECKPOINT_03: Found service-specific run.sh for: web
[DEBUG] Tracking substep start: steps/01-install/steps/web (RUN_UUID=4b33b2ad-c806-4665-b381-eaab322ba3f8)
[INFO] 🔸 Service: web (service-specific)
🔍 DEBUG_CHECKPOINT_04: Executing service-specific: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Web Service Schema Provisioning
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Service:     web
[INFO]   Zone:        universe
[INFO]   Branch:      main
[INFO]   Environment: dev
[INFO]   Identifier:  coordinator
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] 📦 Running: 01-init-schema.sh...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing config schema
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Identifier:  coordinator
  Database:    fastorder_web_universe_main_dev_db
  Host:        db-web-universe-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking if config schema exists...
[INFO] Creating config schema and tables...
CREATE EXTENSION
CREATE SCHEMA
CREATE TABLE
COMMENT
CREATE TABLE
COMMENT
CREATE TABLE
COMMENT
CREATE INDEX
CREATE INDEX
CREATE INDEX
INSERT 0 1
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
NOTICE:  trigger "trg_public_defaults_version" for relation "config.public_defaults" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
NOTICE:  trigger "trg_feature_flags_version" for relation "config.feature_flags" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
NOTICE:  trigger "trg_public_defaults_set_updated_at" for relation "config.public_defaults" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
NOTICE:  trigger "trg_feature_flags_set_updated_at" for relation "config.feature_flags" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
[OK] Config schema and tables created
[INFO] Seeding initial guest services data...
INSERT 0 9
INSERT 0 1
[OK] Initial data seeded
[INFO] Verifying config schema...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Config Schema Verification
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Guest services: 9
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] Config schema initialization complete
[OK] ✓ 01-init-schema.sh completed
[INFO] ═══════════════════════════════════════════════════════════════
[OK] Web service schema provisioning complete
[INFO] ═══════════════════════════════════════════════════════════════

[0;32m✓[0m ✅ Coordinator setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up 1 worker(s) (Citus data nodes)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up worker: worker-01
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-02-05 05:45:02 UTC] USER=unknown EUID=33 PID=3935354 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 05:45:02 UTC] USER=unknown EUID=33 PID=3935365 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 05:45:02 UTC] USER=unknown EUID=33 PID=3935379 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-02-05 05:45:02 UTC] USER=unknown EUID=33 PID=3935404 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 05:45:02 UTC] USER=unknown EUID=33 PID=3935423 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for web in universe-dev...
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=4b33b2ad-c806-4665-b381-eaab322ba3f8 JOB_UUID=582390f3-f4ed-4901-a786-456f7c46742a

[2026-02-05 05:45:03 UTC] USER=unknown EUID=33 PID=3935446 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=4b33b2ad-c806-4665-b381-eaab322ba3f8)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.243
[0;34m[INFO][0m Primary hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com

[0;34m[INFO][0m Adding /etc/hosts entry for worker-01...
[0;34m[INFO][0m   db-web-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.243

[0;34m[INFO][0m   ✅ db-web-universe-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;32m✅[0m   ✅ Network & DNS configuration complete
[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
  10.100.1.243    db-web-universe-main-dev-postgresql-worker-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=4b33b2ad-c806-4665-b381-eaab322ba3f8)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-web-universe-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      web-universe-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01 and /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:45:09 UTC] USER=www-data EUID=0 PID=3937264 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-02-05 05:45:09 UTC] USER=www-data EUID=0 PID=3937365 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-3936664/ra_root.crt
[2026-02-05 05:45:10 UTC] USER=www-data EUID=0 PID=3937394 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-3936664/ra_root.key
[2026-02-05 05:45:10 UTC] USER=www-data EUID=0 PID=3937420 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3936664/ra_root.crt
[2026-02-05 05:45:10 UTC] USER=www-data EUID=0 PID=3937445 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3936664/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[2026-02-05 05:45:16 UTC] USER=www-data EUID=0 PID=3939402 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3936664/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 05:45:16 UTC] USER=www-data EUID=0 PID=3939466 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3936664/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 05:45:16 UTC] USER=www-data EUID=0 PID=3939543 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-02-05 05:45:16 UTC] USER=www-data EUID=0 PID=3939651 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3936664/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:45:16 UTC] USER=www-data EUID=0 PID=3939755 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 05:45:16 UTC] USER=www-data EUID=0 PID=3939780 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 05:45:16 UTC] USER=www-data EUID=0 PID=3939800 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 05:45:16 UTC] USER=www-data EUID=0 PID=3939831 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 05:45:16 UTC] USER=www-data EUID=0 PID=3939893 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 05:45:17 UTC] USER=www-data EUID=0 PID=3939926 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:45:17 UTC] USER=www-data EUID=0 PID=3940019 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: web-universe-main-dev
Node:        worker-01
Primary CN:  db-web-universe-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-web-universe-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 05:45:18 UTC] USER=www-data EUID=0 PID=3940339 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-02-05 05:45:18 UTC] USER=www-data EUID=0 PID=3940357 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 05:45:18 UTC] USER=www-data EUID=0 PID=3940383 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-02-05 05:45:18 UTC] USER=www-data EUID=0 PID=3940414 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:45:18 UTC] USER=www-data EUID=0 PID=3940617 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:45:19 UTC] USER=www-data EUID=0 PID=3940658 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:45:19 UTC] USER=www-data EUID=0 PID=3940712 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 05:45:19 UTC] USER=www-data EUID=0 PID=3940743 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 05:45:19 UTC] USER=www-data EUID=0 PID=3940775 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:45:19 UTC] USER=www-data EUID=0 PID=3940794 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:45:19 UTC] USER=www-data EUID=0 PID=3940835 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 05:45:19 UTC] USER=www-data EUID=0 PID=3940867 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 05:45:19 UTC] USER=www-data EUID=0 PID=3940883 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 05:45:19 UTC] USER=www-data EUID=0 PID=3940899 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:45:19 UTC] USER=www-data EUID=0 PID=3940946 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 05:45:19 UTC] USER=www-data EUID=0 PID=3940969 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3940999 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3941029 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3941049 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3941070 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3941098 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3941107 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3941125 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3941136 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3941147 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3941165 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3941174 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3941183 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3941192 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3941201 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3941211 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3941250 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3941259 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:20 UTC] USER=www-data EUID=0 PID=3941270 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941281 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941292 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941302 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941311 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941329 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941340 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941359 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941368 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941378 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941387 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941396 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941406 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941415 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941424 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941433 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941451 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941463 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941483 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941492 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941501 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941529 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941538 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941547 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941556 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:45:21 UTC] USER=www-data EUID=0 PID=3941565 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 05:45:22 UTC] USER=www-data EUID=0 PID=3941575 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 05:45:22 UTC] USER=www-data EUID=0 PID=3941585 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 05:45:22 UTC] USER=www-data EUID=0 PID=3941595 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:45:22 UTC] USER=www-data EUID=0 PID=3941605 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:45:22 UTC] USER=www-data EUID=0 PID=3941614 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:45:22 UTC] USER=www-data EUID=0 PID=3941627 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 05:45:22 UTC] USER=www-data EUID=0 PID=3941637 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 05:45:22 UTC] USER=www-data EUID=0 PID=3941646 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 05:45:22 UTC] USER=www-data EUID=0 PID=3941655 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:45:22 UTC] USER=www-data EUID=0 PID=3941665 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:45:22 UTC] USER=www-data EUID=0 PID=3941688 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:45:22 UTC] USER=www-data EUID=0 PID=3941697 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: postgres
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 05:45:23 UTC] USER=www-data EUID=0 PID=3941746 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-02-05 05:45:23 UTC] USER=www-data EUID=0 PID=3941774 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 05:45:23 UTC] USER=www-data EUID=0 PID=3941785 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-02-05 05:45:23 UTC] USER=www-data EUID=0 PID=3941795 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 05:45:23 UTC] USER=www-data EUID=0 PID=3941805 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:45:23 UTC] USER=www-data EUID=0 PID=3941823 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:45:23 UTC] USER=www-data EUID=0 PID=3941834 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:45:23 UTC] USER=www-data EUID=0 PID=3941855 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 05:45:23 UTC] USER=www-data EUID=0 PID=3941864 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 05:45:23 UTC] USER=www-data EUID=0 PID=3941892 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:45:23 UTC] USER=www-data EUID=0 PID=3941906 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 05:45:23 UTC] USER=www-data EUID=0 PID=3941916 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3941938 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3941951 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3941960 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3941969 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3941980 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3941991 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3942002 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3942011 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3942020 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3942038 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3942047 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3942075 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3942084 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3942093 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3942122 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3942132 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3942142 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3942151 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3942160 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3942170 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3942179 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3942189 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:24 UTC] USER=www-data EUID=0 PID=3942199 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942208 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942217 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942226 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942235 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942244 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942253 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942281 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942299 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942309 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942328 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942337 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942346 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942355 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942364 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942373 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942382 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942400 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942410 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942419 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942429 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942449 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942458 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:45:25 UTC] USER=www-data EUID=0 PID=3942469 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:45:26 UTC] USER=www-data EUID=0 PID=3942490 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 05:45:26 UTC] USER=www-data EUID=0 PID=3942504 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 05:45:26 UTC] USER=www-data EUID=0 PID=3942514 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:45:26 UTC] USER=www-data EUID=0 PID=3942523 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:45:26 UTC] USER=www-data EUID=0 PID=3942532 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 05:45:26 UTC] USER=www-data EUID=0 PID=3942543 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 05:45:26 UTC] USER=www-data EUID=0 PID=3942560 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 05:45:26 UTC] USER=www-data EUID=0 PID=3942570 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:45:26 UTC] USER=www-data EUID=0 PID=3942594 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:45:26 UTC] USER=www-data EUID=0 PID=3942604 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 05:45:26 UTC] USER=www-data EUID=0 PID=3942614 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 05:45:26 UTC] USER=www-data EUID=0 PID=3942625 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 05:45:26 UTC] USER=www-data EUID=0 PID=3942636 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:45:26 UTC] USER=www-data EUID=0 PID=3942647 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:45:26 UTC] USER=www-data EUID=0 PID=3942656 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:45:26 UTC] USER=www-data EUID=0 PID=3942665 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: postgres
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=4b33b2ad-c806-4665-b381-eaab322ba3f8)
[INFO] 📦 02 setup pg instance...
[0;34m[DEADLOCK-PREVENTION][0m Deadlock prevention library loaded
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-postgresql environment: db-web-universe-main-dev-postgresql-worker-01.fastorder.com (10.100.1.243)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.243
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m Data dir:   /data/postgresql/17/web-universe-main-dev/worker-01
[0;34m[INFO][0m Port:       5432
[0;34m[INFO][0m Hostname:   db-web-universe-main-dev-postgresql-worker-01
[2026-02-05 05:45:28 UTC] USER=www-data EUID=0 PID=3942825 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:45:28 UTC] USER=www-data EUID=0 PID=3942883 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[1;33m[WARN][0m Server certificate not found at /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[0;34m[INFO][0m Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-web-universe-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      web-universe-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 05:45:29 UTC] USER=www-data EUID=0 PID=3942936 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01 and /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:45:29 UTC] USER=www-data EUID=0 PID=3942945 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-02-05 05:45:29 UTC] USER=www-data EUID=0 PID=3942955 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-3942890
[2026-02-05 05:45:29 UTC] USER=www-data EUID=0 PID=3942964 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-3942890/ra_root.crt
[2026-02-05 05:45:29 UTC] USER=www-data EUID=0 PID=3942974 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-3942890/ra_root.key
[2026-02-05 05:45:29 UTC] USER=www-data EUID=0 PID=3942983 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3942890/ra_root.crt
[2026-02-05 05:45:29 UTC] USER=www-data EUID=0 PID=3942993 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3942890/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[2026-02-05 05:45:30 UTC] USER=www-data EUID=0 PID=3943096 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3942890/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 05:45:30 UTC] USER=www-data EUID=0 PID=3943105 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-02-05 05:45:30 UTC] USER=www-data EUID=0 PID=3943114 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3942890/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:45:30 UTC] USER=www-data EUID=0 PID=3943123 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:45:31 UTC] USER=www-data EUID=0 PID=3943137 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:45:31 UTC] USER=www-data EUID=0 PID=3943158 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 05:45:31 UTC] USER=www-data EUID=0 PID=3943182 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 05:45:31 UTC] USER=www-data EUID=0 PID=3943192 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 05:45:31 UTC] USER=www-data EUID=0 PID=3943206 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 05:45:31 UTC] USER=www-data EUID=0 PID=3943216 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 05:45:31 UTC] USER=www-data EUID=0 PID=3943225 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: web-universe-main-dev
Node:        worker-01
Primary CN:  db-web-universe-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-web-universe-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-02-05 05:45:31 UTC] USER=www-data EUID=0 PID=3943272 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 05:45:31 UTC] USER=www-data EUID=0 PID=3943290 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 05:45:31 UTC] USER=www-data EUID=0 PID=3943299 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[0;32m[OK][0m   mTLS certificates OK (server cert + client certs verified) and keys secured
[0;34m[INFO][0m Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-02-05 05:45:32 UTC] USER=www-data EUID=0 PID=3943320 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 05:45:32 UTC] USER=www-data EUID=0 PID=3943348 ACTION=passthru ARGS=systemctl stop postgresql
[1;33m[WARN][0m Cleaning stale socket directory /var/run/postgresql-web-universe-main-dev-worker-01
[2026-02-05 05:45:32 UTC] USER=www-data EUID=0 PID=3943382 ACTION=fsop ARGS=rm -rf /var/run/postgresql-web-universe-main-dev-worker-01
[0;32m[OK][0m   No conflicting Postgres left on port 5432
[0;32m[OK][0m   Using postgres password from vault provider
[2026-02-05 05:45:35 UTC] USER=www-data EUID=0 PID=3943533 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.YC11Dr
[2026-02-05 05:45:35 UTC] USER=www-data EUID=0 PID=3943554 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.YC11Dr
[2026-02-05 05:45:35 UTC] USER=www-data EUID=0 PID=3943578 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev
[2026-02-05 05:45:35 UTC] USER=www-data EUID=0 PID=3943600 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev
[2026-02-05 05:45:35 UTC] USER=www-data EUID=0 PID=3943631 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/web-universe-main-dev
[0;34m[INFO][0m Initializing cluster in /data/postgresql/17/web-universe-main-dev/worker-01 (SCRAM; pwfile)
[1;33m[WARN][0m Removing existing data directory: /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 05:45:36 UTC] USER=www-data EUID=0 PID=3943653 ACTION=fsop ARGS=rm -rf /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 05:45:36 UTC] USER=www-data EUID=0 PID=3943686 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 05:45:36 UTC] USER=www-data EUID=0 PID=3943711 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 05:45:36 UTC] USER=www-data EUID=0 PID=3943740 ACTION=fsop ARGS=chmod 700 /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 05:45:36 UTC] USER=www-data EUID=0 PID=3943786 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-web-universe-main-dev-worker-01
[2026-02-05 05:45:36 UTC] USER=www-data EUID=0 PID=3943810 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-web-universe-main-dev-worker-01
[2026-02-05 05:45:37 UTC] USER=www-data EUID=0 PID=3943826 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/web-universe-main-dev/worker-01 --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.YC11Dr
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /data/postgresql/17/web-universe-main-dev/worker-01 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/web-universe-main-dev/worker-01 -l logfile start

[0;32m[OK][0m   initdb complete
[2026-02-05 05:45:38 UTC] USER=www-data EUID=0 PID=3943913 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.YC11Dr
[0;34m[INFO][0m Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[0;32m[OK][0m   postgresql.conf updated successfully
[0;34m[INFO][0m Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-02-05 05:45:38 UTC] USER=www-data EUID=0 PID=3943967 ACTION=fsop ARGS=cp /tmp/tmp.rTWhgddy5o /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
[2026-02-05 05:45:38 UTC] USER=www-data EUID=0 PID=3943988 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
[2026-02-05 05:45:38 UTC] USER=www-data EUID=0 PID=3944010 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
[0;32m[OK][0m   pg_hba.conf updated
[0;34m[INFO][0m Creating systemd unit: /etc/systemd/system/postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 05:45:38 UTC] USER=www-data EUID=0 PID=3944048 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.IHdsua /etc/systemd/system/postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 05:45:38 UTC] USER=www-data EUID=0 PID=3944084 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@web-universe-main-dev-worker-01.service
[0;32m[OK][0m   systemd unit written
[2026-02-05 05:45:39 UTC] USER=www-data EUID=0 PID=3944144 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 05:45:39 UTC] USER=www-data EUID=0 PID=3944165 ACTION=passthru ARGS=systemctl daemon-reload
[0;34m[INFO][0m Starting PostgreSQL instance...
[2026-02-05 05:45:40 UTC] USER=www-data EUID=0 PID=3944299 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-worker-01.service
[0;34m[INFO][0m Waiting for ACTIVE (systemd)…
[2026-02-05 05:45:41 UTC] USER=www-data EUID=0 PID=3944346 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-worker-01.service
[0;32m[OK][0m   Service ACTIVE
[0;34m[INFO][0m Waiting for port 5432 bind…
[0;32m[OK][0m   Port bound
[0;34m[INFO][0m Waiting pg_isready (socket)…
[0;32m[OK][0m   Readiness via socket OK
[0;34m[INFO][0m Waiting pg_isready (TCP db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432)…
[0;32m[OK][0m   Startup sequence complete
[0;34m[INFO][0m Validating core security GUCs (via local socket)…
[0;32m[OK][0m   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[0;34m[INFO][0m Provisioning application database and Debezium role (if not exists)...
[0;34m[INFO][0m Checking if database fastorder_web_universe_main_dev_db exists...
[0;34m[INFO][0m DB check result: exit_code=0, output='[2026-02-05 05:45:42 UTC] USER=www-data EUID=0 PID=3944521 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_web_universe_main_dev_db''
[0;34m[INFO][0m Creating database fastorder_web_universe_main_dev_db...
[2026-02-05 05:45:42 UTC] USER=www-data EUID=0 PID=3944554 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_web_universe_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[0;32m[OK][0m   Database fastorder_web_universe_main_dev_db created
[0;34m[INFO][0m Checking if role debezium_user exists...
[0;34m[INFO][0m Role check result: exit_code=0, output='[2026-02-05 05:45:42 UTC] USER=www-data EUID=0 PID=3944578 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[0;34m[INFO][0m Creating role debezium_user...
[2026-02-05 05:45:42 UTC] USER=www-data EUID=0 PID=3944616 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'OW7JQhJo6X6jPZS9JkNw65yv';
CREATE ROLE
[0;32m[OK][0m   Role debezium_user created
[2026-02-05 05:45:43 UTC] USER=www-data EUID=0 PID=3944641 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_web_universe_main_dev_db" TO debezium_user;
GRANT
[0;32m[OK][0m   Application DB (fastorder_web_universe_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[0;34m[INFO][0m Applying connection and memory optimizations...
[0;34m[INFO][0m Current settings: max_connections=100, work_mem=4MB
[0;34m[INFO][0m Target settings (worker): max_connections=100, work_mem=8MB
[2026-02-05 05:45:43 UTC] USER=www-data EUID=0 PID=3944724 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 100;
ALTER SYSTEM
[2026-02-05 05:45:43 UTC] USER=www-data EUID=0 PID=3944753 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-02-05 05:45:43 UTC] USER=www-data EUID=0 PID=3944776 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[0;32m[OK][0m   Settings applied to postgresql.auto.conf
[2026-02-05 05:45:43 UTC] USER=www-data EUID=0 PID=3944791 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/worker-01/standby.signal
[0;34m[INFO][0m Service recently started (2s ago) - restarting to apply max_connections...
[0;34m[INFO][0m Stopping service...
[2026-02-05 05:45:43 UTC] USER=www-data EUID=0 PID=3944816 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-worker-01.service
[0;34m[INFO][0m Waiting for port 5432 to be released...
[0;32m[OK][0m   Port 5432 released
[0;34m[INFO][0m Starting service...
[2026-02-05 05:45:47 UTC] USER=www-data EUID=0 PID=3944986 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 05:45:53 UTC] USER=www-data EUID=0 PID=3945213 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-worker-01.service
[0;32m[OK][0m   ✅ Optimization complete: max_connections=100, work_mem=8MB
[0;32m[OK][0m   Synchronous replication already configured (synchronous_commit: on)
[0;34m[INFO][0m Setting postgres password via centralized script... for worker-01
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    web[0m
[0;34m  Zone:       universe[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: worker-01[0m

[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-web-universe-main-dev-worker-01[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mInitial setup: Using password from initdb[0m
[0;32m✓ PostgreSQL password already set during initdb[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/worker-01
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/worker-01[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials worker-01[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
[0;32m[OK][0m   Password set and persisted
[0;34m[INFO][0m Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.243
[0;34m[INFO][0m Primary hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com

[0;34m[INFO][0m Adding /etc/hosts entry for worker-01...
[0;34m[INFO][0m   db-web-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.243

[0;34m[INFO][0m   ✅ db-web-universe-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;32m✅[0m   ✅ Network & DNS configuration complete
[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
  10.100.1.243    db-web-universe-main-dev-postgresql-worker-01.fastorder.com


[0;32m[OK][0m   PostgreSQL 'web-universe-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key \
        host=db-web-universe-main-dev-postgresql-worker-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[0;34m[INFO][0m Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        web-universe-main-dev-postgresql-worker-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.243
[INFO]   Port:              5432
[INFO]   FQDN:              db-web-universe-main-dev-postgresql-worker-01
[INFO]   Status:            running
[INFO]   Environment:       web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 84e4185a-2ef1-49c1-8d2a-841d077f036b
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[0;32m[OK][0m   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=4b33b2ad-c806-4665-b381-eaab322ba3f8)
[INFO] 📦 03 role...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 05:46:06 UTC] USER=www-data EUID=0 PID=3946396 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/worker-01/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    debezium_user
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   debezium_user
  Hostname:    db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 05:46:07 UTC] USER=www-data EUID=0 PID=3946586 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-debezium_user
[2026-02-05 05:46:07 UTC] USER=www-data EUID=0 PID=3946595 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-02-05 05:46:08 UTC] USER=www-data EUID=0 PID=3946619 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-02-05 05:46:08 UTC] USER=www-data EUID=0 PID=3946630 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:46:08 UTC] USER=www-data EUID=0 PID=3946655 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:46:08 UTC] USER=www-data EUID=0 PID=3946671 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:46:08 UTC] USER=www-data EUID=0 PID=3946680 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 05:46:08 UTC] USER=www-data EUID=0 PID=3946689 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 05:46:08 UTC] USER=www-data EUID=0 PID=3946698 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:08 UTC] USER=www-data EUID=0 PID=3946707 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:46:08 UTC] USER=www-data EUID=0 PID=3946716 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 05:46:08 UTC] USER=www-data EUID=0 PID=3946725 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 05:46:08 UTC] USER=www-data EUID=0 PID=3946736 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 05:46:08 UTC] USER=www-data EUID=0 PID=3946754 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:08 UTC] USER=www-data EUID=0 PID=3946763 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3946772 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3946784 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3946797 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3946806 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3946817 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3946827 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3946864 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3946874 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3946883 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3946897 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3946926 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3946940 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3946951 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3946989 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3947000 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3947010 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3947021 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3947042 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3947070 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:09 UTC] USER=www-data EUID=0 PID=3947090 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947114 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947125 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947134 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947152 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947161 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947170 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947182 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947191 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947200 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947210 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947221 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947230 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947239 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947248 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947262 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947287 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947306 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947330 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947349 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:46:10 UTC] USER=www-data EUID=0 PID=3947374 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 05:46:11 UTC] USER=www-data EUID=0 PID=3947395 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 05:46:11 UTC] USER=www-data EUID=0 PID=3947428 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 05:46:11 UTC] USER=www-data EUID=0 PID=3947458 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:11 UTC] USER=www-data EUID=0 PID=3947485 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:11 UTC] USER=www-data EUID=0 PID=3947514 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:11 UTC] USER=www-data EUID=0 PID=3947550 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:11 UTC] USER=www-data EUID=0 PID=3947567 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:11 UTC] USER=www-data EUID=0 PID=3947588 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:11 UTC] USER=www-data EUID=0 PID=3947618 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 05:46:11 UTC] USER=www-data EUID=0 PID=3947669 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:11 UTC] USER=www-data EUID=0 PID=3947687 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:46:11 UTC] USER=www-data EUID=0 PID=3947705 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 05:46:11 UTC] USER=www-data EUID=0 PID=3947726 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 05:46:11 UTC] USER=www-data EUID=0 PID=3947747 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:46:12 UTC] USER=www-data EUID=0 PID=3947815 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:46:12 UTC] USER=www-data EUID=0 PID=3947857 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:46:12 UTC] USER=www-data EUID=0 PID=3947885 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 05:46:12 UTC] USER=www-data EUID=0 PID=3947909 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 05:46:12 UTC] USER=www-data EUID=0 PID=3947927 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 05:46:12 UTC] USER=www-data EUID=0 PID=3947946 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:46:12 UTC] USER=www-data EUID=0 PID=3947968 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:46:12 UTC] USER=www-data EUID=0 PID=3947990 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:46:12 UTC] USER=www-data EUID=0 PID=3948008 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: debezium_user
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U debezium_user -d postgres

🔐 Generating replicator client certificate for worker-01...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 05:46:13 UTC] USER=www-data EUID=0 PID=3948190 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-02-05 05:46:13 UTC] USER=www-data EUID=0 PID=3948202 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-02-05 05:46:13 UTC] USER=www-data EUID=0 PID=3948223 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-02-05 05:46:13 UTC] USER=www-data EUID=0 PID=3948242 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-02-05 05:46:13 UTC] USER=www-data EUID=0 PID=3948269 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:46:13 UTC] USER=www-data EUID=0 PID=3948367 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:46:13 UTC] USER=www-data EUID=0 PID=3948395 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:46:13 UTC] USER=www-data EUID=0 PID=3948461 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:13 UTC] USER=www-data EUID=0 PID=3948487 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:46:14 UTC] USER=www-data EUID=0 PID=3948509 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 05:46:14 UTC] USER=www-data EUID=0 PID=3948534 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 05:46:14 UTC] USER=www-data EUID=0 PID=3948556 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 05:46:14 UTC] USER=www-data EUID=0 PID=3948567 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 05:46:14 UTC] USER=www-data EUID=0 PID=3948576 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 05:46:14 UTC] USER=www-data EUID=0 PID=3948585 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 05:46:14 UTC] USER=www-data EUID=0 PID=3948594 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 05:46:14 UTC] USER=www-data EUID=0 PID=3948613 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:46:14 UTC] USER=www-data EUID=0 PID=3948622 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 05:46:14 UTC] USER=www-data EUID=0 PID=3948631 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 05:46:14 UTC] USER=www-data EUID=0 PID=3948640 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 05:46:14 UTC] USER=www-data EUID=0 PID=3948649 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 05:46:14 UTC] USER=www-data EUID=0 PID=3948661 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:14 UTC] USER=www-data EUID=0 PID=3948686 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:14 UTC] USER=www-data EUID=0 PID=3948726 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:14 UTC] USER=www-data EUID=0 PID=3948737 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:14 UTC] USER=www-data EUID=0 PID=3948748 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948770 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948797 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948807 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948816 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948827 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948840 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948850 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948860 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948870 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948881 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948891 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948901 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948910 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948930 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948939 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948949 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948958 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948967 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948977 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948987 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:15 UTC] USER=www-data EUID=0 PID=3948998 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949010 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949019 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949028 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949037 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949046 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949056 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949065 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949074 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949083 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949093 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949103 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949122 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949140 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949149 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949158 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949167 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949176 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949186 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949195 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949204 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949214 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949226 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949237 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949246 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:46:16 UTC] USER=www-data EUID=0 PID=3949255 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 05:46:17 UTC] USER=www-data EUID=0 PID=3949264 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 05:46:17 UTC] USER=www-data EUID=0 PID=3949278 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 05:46:17 UTC] USER=www-data EUID=0 PID=3949296 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:46:17 UTC] USER=www-data EUID=0 PID=3949305 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:46:17 UTC] USER=www-data EUID=0 PID=3949314 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: replicator
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

✅ Replicator certificate generated for worker-01
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    web[0m
[0;34m  Zone:       universe[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: worker-01[0m

[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-web-universe-main-dev-worker-01[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mGenerating new secure password...[0m
[0;34mUser fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: fastorder_admin_gd[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials worker-01[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: worker-01/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    fastorder_admin_gd
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   fastorder_admin_gd
  Hostname:    db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 05:46:28 UTC] USER=www-data EUID=0 PID=3950169 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-fastorder_admin_gd
[2026-02-05 05:46:28 UTC] USER=www-data EUID=0 PID=3950188 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
[2026-02-05 05:46:28 UTC] USER=www-data EUID=0 PID=3950197 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-02-05 05:46:28 UTC] USER=www-data EUID=0 PID=3950206 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:46:29 UTC] USER=www-data EUID=0 PID=3950232 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:46:29 UTC] USER=www-data EUID=0 PID=3950251 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:46:29 UTC] USER=www-data EUID=0 PID=3950260 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 05:46:29 UTC] USER=www-data EUID=0 PID=3950269 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 05:46:29 UTC] USER=www-data EUID=0 PID=3950278 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:29 UTC] USER=www-data EUID=0 PID=3950288 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:46:29 UTC] USER=www-data EUID=0 PID=3950299 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 05:46:29 UTC] USER=www-data EUID=0 PID=3950308 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 05:46:29 UTC] USER=www-data EUID=0 PID=3950317 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 05:46:29 UTC] USER=www-data EUID=0 PID=3950326 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 05:46:29 UTC] USER=www-data EUID=0 PID=3950337 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 05:46:29 UTC] USER=www-data EUID=0 PID=3950346 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 05:46:29 UTC] USER=www-data EUID=0 PID=3950355 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 05:46:29 UTC] USER=www-data EUID=0 PID=3950368 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:29 UTC] USER=www-data EUID=0 PID=3950389 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 05:46:29 UTC] USER=www-data EUID=0 PID=3950398 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 05:46:29 UTC] USER=www-data EUID=0 PID=3950407 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950417 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950427 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950436 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950445 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950472 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950481 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950490 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950499 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950508 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950517 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950526 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950535 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950544 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950555 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950568 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950577 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950588 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950598 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950607 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950616 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950625 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950634 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:30 UTC] USER=www-data EUID=0 PID=3950643 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950652 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950661 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950674 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950687 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950697 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950716 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950726 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950736 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950745 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950754 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950763 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950772 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950781 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950790 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950801 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950811 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950822 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950831 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950843 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950862 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950871 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:46:31 UTC] USER=www-data EUID=0 PID=3950880 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:32 UTC] USER=www-data EUID=0 PID=3950889 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 05:46:32 UTC] USER=www-data EUID=0 PID=3950900 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 05:46:32 UTC] USER=www-data EUID=0 PID=3950909 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 05:46:32 UTC] USER=www-data EUID=0 PID=3950919 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 05:46:32 UTC] USER=www-data EUID=0 PID=3950928 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 05:46:32 UTC] USER=www-data EUID=0 PID=3950937 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 05:46:32 UTC] USER=www-data EUID=0 PID=3950948 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 05:46:32 UTC] USER=www-data EUID=0 PID=3950957 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 05:46:32 UTC] USER=www-data EUID=0 PID=3950967 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:46:32 UTC] USER=www-data EUID=0 PID=3950977 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:46:32 UTC] USER=www-data EUID=0 PID=3950986 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:46:32 UTC] USER=www-data EUID=0 PID=3950995 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 05:46:32 UTC] USER=www-data EUID=0 PID=3951004 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
   ✅ Symlinked client-key.pem
[2026-02-05 05:46:32 UTC] USER=www-data EUID=0 PID=3951022 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:46:32 UTC] USER=www-data EUID=0 PID=3951031 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:46:32 UTC] USER=www-data EUID=0 PID=3951040 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:46:32 UTC] USER=www-data EUID=0 PID=3951050 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: fastorder_admin_gd
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-web-universe-main-dev-worker-01:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
📦 Creating database fastorder_web_universe_main_dev_db...
[2026-02-05 05:46:33 UTC] USER=www-data EUID=0 PID=3951120 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
ERROR:  database "fastorder_web_universe_main_dev_db" already exists
[2026-02-05 05:46:33 UTC] USER=www-data EUID=0 PID=3951145 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-02-05 05:46:33 UTC] USER=www-data EUID=0 PID=3951183 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-worker-01.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

[0;34m=== Pre-flight Checks ===[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;32m✓[0m AWS Secrets Manager accessible

[0;34m=== Retrieving Credentials from AWS ===[0m
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials loaded for worker-01/fastorder_admin_gd: fastorder_admin_gd@db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_web_universe_main_dev_db
[0;32m✓[0m Credentials retrieved: fastorder_admin_gd@db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_web_universe_main_dev_db
[0;34m╔════════════════════════════════════════════╗[0m
[0;34m║  PostgreSQL Test Suite (AWS Secrets MGR)  ║[0m
[0;34m╚════════════════════════════════════════════╝[0m

[0;34m=== PostgreSQL Authentication Test ===[0m
[0;31m✗[0m PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-web-universe-main-dev-postgresql-worker-01.fastorder.com" (10.100.1.243), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 05:46:39 UTC] USER=www-data EUID=0 PID=3951646 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/worker-01/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : web-universe-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_web_universe_main_dev_db
  SCHEMA      : web
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
  DNS → 10.100.1.243
  CA         : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    web[0m
[0;34m  Zone:       universe[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: worker-01[0m

[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-web-universe-main-dev-worker-01[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mGenerating new secure password...[0m
[0;34mUser debezium_user does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: debezium_user[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials worker-01[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: worker-01/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-02-05 05:46:47 UTC] USER=www-data EUID=0 PID=3952150 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_web_universe_main_dev_db already exists
[2026-02-05 05:46:47 UTC] USER=www-data EUID=0 PID=3952181 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d fastorder_web_universe_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
🔐 Creating replicator role for worker-01...
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : web-universe-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SSL DIR     : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
  DNS → 10.100.1.243
  CA         : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Creating role: replicator with password
SET
CREATE ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/web/universe/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🎉 Done.
✅ Replicator role created for worker-01

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=4b33b2ad-c806-4665-b381-eaab322ba3f8)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (web) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=web
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
🔍 DEBUG_CHECKPOINT_03: Found service-specific run.sh for: web
[DEBUG] Tracking substep start: steps/01-install/steps/web (RUN_UUID=4b33b2ad-c806-4665-b381-eaab322ba3f8)
[INFO] 🔸 Service: web (service-specific)
🔍 DEBUG_CHECKPOINT_04: Executing service-specific: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Web Service Schema Provisioning
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Service:     web
[INFO]   Zone:        universe
[INFO]   Branch:      main
[INFO]   Environment: dev
[INFO]   Identifier:  coordinator
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] 📦 Running: 01-init-schema.sh...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing config schema
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Identifier:  coordinator
  Database:    fastorder_web_universe_main_dev_db
  Host:        db-web-universe-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking if config schema exists...
[INFO] Config schema already exists - checking tables...
[OK] Config schema with 3 tables already exists - skipping
[OK] ✓ 01-init-schema.sh completed
[INFO] ═══════════════════════════════════════════════════════════════
[OK] Web service schema provisioning complete
[INFO] ═══════════════════════════════════════════════════════════════

[0;32m✓[0m ✅ Worker worker-01 setup completed

Skipping standbys (PG_WORKERS_STANDBY_NUM=0)
[0;32m✓[0m ✅ PostgreSQL installation completed
[0;34m[INFO][0m Discovering additional setup steps...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 02-pg-bouncer.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up PgBouncer connection pooling...
[2026-02-05 05:46:56 UTC] USER=www-data EUID=0 PID=3952730 ACTION=fsop ARGS=rm -f /tmp/pgbouncer-ip.service /tmp/pgbouncer.service
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;32m✓ [SECRETS][0m Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[0;34m[SECRETS][0m Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[0;34m[SECRETS][0m            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[0;34m[SECRETS][0m            Backups (build_backup_path)
[0;34m[SECRETS][0m Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[0;34m[INFO][0m Checking for existing PgBouncer application environment in topology …
[0;32m[OK][0m   Using existing PgBouncer environment:
[0;34m[INFO][0m   IP:     10.100.1.244
[0;34m[INFO][0m   FQDN:   db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[0;34m[INFO][0m   Domain: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[0;34m[INFO][0m Ensuring /etc/hosts entry for db-web-universe-main-dev-postgresql-bouncer.fastorder.com …
[0;32m[OK][0m   /etc/hosts already contains entry for db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[1;33m[WARN][0m IP 10.100.1.244 is assigned to multiple interfaces:
    inet 10.100.1.242/32 scope global lo
       valid_lft forever preferred_lft forever
    inet 10.100.1.244/32 scope global lo:pgbouncer
--
    inet 10.100.1.243/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.100.1.244/32 scope global eth0:244
[1;33m[WARN][0m This may cause routing issues
[0;34m[INFO][0m Final verification of /etc/hosts entry for db-web-universe-main-dev-postgresql-bouncer.fastorder.com …
[0;32m[OK][0m   /etc/hosts correctly maps db-web-universe-main-dev-postgresql-bouncer.fastorder.com to 10.100.1.244
[0;32m[OK][0m   PgBouncer IP 10.100.1.244 already correctly bound to lo:pgbouncer
[2026-02-05 05:46:57 UTC] USER=www-data EUID=0 PID=3952911 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 05:46:58 UTC] USER=www-data EUID=0 PID=3953227 ACTION=passthru ARGS=systemctl restart pgbouncer-ip@web-universe-main-dev.service
Job for pgbouncer-ip@web-universe-main-dev.service failed because the control process exited with error code.
See "systemctl status pgbouncer-ip@web-universe-main-dev.service" and "journalctl -xeu pgbouncer-ip@web-universe-main-dev.service" for details.
[2026-02-05 05:46:58 UTC] USER=www-data EUID=0 PID=3953262 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer-ip@web-universe-main-dev.service
[1;33m[WARN][0m pgbouncer-ip@web-universe-main-dev.service is not active
[1;33m[WARN][0m Check status: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@web-universe-main-dev.service
[2026-02-05 05:46:58 UTC] USER=www-data EUID=0 PID=3953381 ACTION=fsop ARGS=mkdir -p /etc/pgbouncer/web-universe-main-dev
[2026-02-05 05:46:58 UTC] USER=www-data EUID=0 PID=3953397 ACTION=fsop ARGS=mkdir -p /run/pgbouncer/web-universe-main-dev
[2026-02-05 05:46:58 UTC] USER=www-data EUID=0 PID=3953418 ACTION=fsop ARGS=mkdir -p /var/log/pgbouncer/web-universe-main-dev
[2026-02-05 05:46:58 UTC] USER=www-data EUID=0 PID=3953435 ACTION=fsop ARGS=chmod 750 /etc/pgbouncer/web-universe-main-dev
[2026-02-05 05:46:58 UTC] USER=www-data EUID=0 PID=3953452 ACTION=fsop ARGS=chmod 750 /run/pgbouncer/web-universe-main-dev
[2026-02-05 05:46:58 UTC] USER=www-data EUID=0 PID=3953470 ACTION=fsop ARGS=chmod 750 /var/log/pgbouncer/web-universe-main-dev
[2026-02-05 05:46:58 UTC] USER=www-data EUID=0 PID=3953488 ACTION=fsop ARGS=chown root:postgres /etc/pgbouncer/web-universe-main-dev
[2026-02-05 05:46:58 UTC] USER=www-data EUID=0 PID=3953505 ACTION=fsop ARGS=chown postgres:postgres /run/pgbouncer/web-universe-main-dev
[2026-02-05 05:46:58 UTC] USER=www-data EUID=0 PID=3953526 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbouncer/web-universe-main-dev
[0;34m[INFO][0m Generating PgBouncer TLS certificate via existing server.sh (IDENTIFIER=pgbouncer) …
[0;34m[INFO][0m Calling: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/ssl/server.sh pgbouncer
[0;34m[INFO][0m With: OVERRIDE_CN=db-web-universe-main-dev-postgresql-bouncer.fastorder.com, OVERRIDE_IP=10.100.1.244
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        pgbouncer
  Primary CN:  db-web-universe-main-dev-postgresql-bouncer.fastorder.com
  Alt CN:      web-universe-main-dev.fastorder.com
  VM IP:       10.100.1.244
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 05:46:59 UTC] USER=www-data EUID=0 PID=3953718 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer and /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 05:46:59 UTC] USER=www-data EUID=0 PID=3953753 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
🔐 Generating 4096-bit private key...
[2026-02-05 05:46:59 UTC] USER=www-data EUID=0 PID=3953778 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-3953541
[2026-02-05 05:46:59 UTC] USER=www-data EUID=0 PID=3953793 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-3953541/ra_root.crt
[2026-02-05 05:46:59 UTC] USER=www-data EUID=0 PID=3953823 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-3953541/ra_root.key
[2026-02-05 05:46:59 UTC] USER=www-data EUID=0 PID=3953852 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3953541/ra_root.crt
[2026-02-05 05:46:59 UTC] USER=www-data EUID=0 PID=3953879 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3953541/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[2026-02-05 05:47:01 UTC] USER=www-data EUID=0 PID=3954151 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3953541/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
[2026-02-05 05:47:01 UTC] USER=www-data EUID=0 PID=3954169 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3953541/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
[2026-02-05 05:47:01 UTC] USER=www-data EUID=0 PID=3954187 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
📋 Setting up CA certificate...
[2026-02-05 05:47:01 UTC] USER=www-data EUID=0 PID=3954223 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 05:47:01 UTC] USER=www-data EUID=0 PID=3954241 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 05:47:01 UTC] USER=www-data EUID=0 PID=3954250 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 05:47:01 UTC] USER=www-data EUID=0 PID=3954262 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
[2026-02-05 05:47:01 UTC] USER=www-data EUID=0 PID=3954271 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
[2026-02-05 05:47:01 UTC] USER=www-data EUID=0 PID=3954280 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
[2026-02-05 05:47:01 UTC] USER=www-data EUID=0 PID=3954303 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:01 UTC] USER=www-data EUID=0 PID=3954312 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-bouncer.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-web-universe-main-dev-postgresql-bouncer.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-bouncer.fastorder.com, DNS:db-web-universe-main-dev-postgresql-bouncer, DNS:localhost, IP Address:10.100.1.244, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: web-universe-main-dev
Node:        pgbouncer
Primary CN:  db-web-universe-main-dev-postgresql-bouncer.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt'

2. Restart PgBouncer:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@web-universe-main-dev.service

3. Test SSL connection:
   psql "host=db-web-universe-main-dev-postgresql-bouncer.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m   mTLS server certificate present: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
[0;34m[INFO][0m Generating pgbouncer_admin client certificates...
[0;34m[INFO][0m ⏳ This may take 30-60 seconds...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username:    pgbouncer_admin
Identifier:  pgbouncer
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: web-universe-main-dev
  Service:     web
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        pgbouncer
  User (CN):   pgbouncer_admin
  Hostname:    db-web-universe-main-dev-postgresql-bouncer.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 05:47:02 UTC] USER=www-data EUID=0 PID=3954527 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-pgbouncer-pgbouncer_admin
[2026-02-05 05:47:02 UTC] USER=www-data EUID=0 PID=3954573 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
[2026-02-05 05:47:02 UTC] USER=www-data EUID=0 PID=3954593 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-02-05 05:47:02 UTC] USER=www-data EUID=0 PID=3954607 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = pgbouncer_admin
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3954786 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3954805 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3954819 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3954840 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3954860 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3954882 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3954901 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3954920 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3954938 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3954951 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3954968 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3954985 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3955008 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3955027 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3955048 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3955063 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3955085 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3955104 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3955152 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3955168 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:47:03 UTC] USER=www-data EUID=0 PID=3955201 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:47:04 UTC] USER=www-data EUID=0 PID=3955216 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:04 UTC] USER=www-data EUID=0 PID=3955248 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:04 UTC] USER=www-data EUID=0 PID=3955278 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 05:47:04 UTC] USER=www-data EUID=0 PID=3955299 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 05:47:04 UTC] USER=www-data EUID=0 PID=3955317 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 05:47:04 UTC] USER=www-data EUID=0 PID=3955341 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 05:47:04 UTC] USER=www-data EUID=0 PID=3955360 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 05:47:04 UTC] USER=www-data EUID=0 PID=3955390 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 05:47:04 UTC] USER=www-data EUID=0 PID=3955418 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 05:47:04 UTC] USER=www-data EUID=0 PID=3955439 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:04 UTC] USER=www-data EUID=0 PID=3955476 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:04 UTC] USER=www-data EUID=0 PID=3955498 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:47:04 UTC] USER=www-data EUID=0 PID=3955539 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:04 UTC] USER=www-data EUID=0 PID=3955564 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:04 UTC] USER=www-data EUID=0 PID=3955580 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 05:47:05 UTC] USER=www-data EUID=0 PID=3955598 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 05:47:05 UTC] USER=www-data EUID=0 PID=3955626 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 05:47:05 UTC] USER=www-data EUID=0 PID=3955654 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 05:47:05 UTC] USER=www-data EUID=0 PID=3955685 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 05:47:05 UTC] USER=www-data EUID=0 PID=3955702 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 05:47:05 UTC] USER=www-data EUID=0 PID=3955724 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 05:47:05 UTC] USER=www-data EUID=0 PID=3955747 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:05 UTC] USER=www-data EUID=0 PID=3955758 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:05 UTC] USER=www-data EUID=0 PID=3955767 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:47:05 UTC] USER=www-data EUID=0 PID=3955776 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:47:05 UTC] USER=www-data EUID=0 PID=3955785 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:05 UTC] USER=www-data EUID=0 PID=3955795 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:05 UTC] USER=www-data EUID=0 PID=3955804 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 05:47:05 UTC] USER=www-data EUID=0 PID=3955813 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 05:47:05 UTC] USER=www-data EUID=0 PID=3955823 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 05:47:05 UTC] USER=www-data EUID=0 PID=3955837 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 05:47:05 UTC] USER=www-data EUID=0 PID=3955852 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 05:47:06 UTC] USER=www-data EUID=0 PID=3955865 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 05:47:06 UTC] USER=www-data EUID=0 PID=3955897 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 05:47:06 UTC] USER=www-data EUID=0 PID=3955912 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:06 UTC] USER=www-data EUID=0 PID=3955925 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:06 UTC] USER=www-data EUID=0 PID=3955935 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:47:06 UTC] USER=www-data EUID=0 PID=3955947 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 05:47:06 UTC] USER=www-data EUID=0 PID=3955960 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:06 UTC] USER=www-data EUID=0 PID=3955971 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:06 UTC] USER=www-data EUID=0 PID=3955990 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 05:47:07 UTC] USER=www-data EUID=0 PID=3956001 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 05:47:07 UTC] USER=www-data EUID=0 PID=3956010 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 05:47:07 UTC] USER=www-data EUID=0 PID=3956019 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 05:47:07 UTC] USER=www-data EUID=0 PID=3956028 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 05:47:07 UTC] USER=www-data EUID=0 PID=3956037 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 05:47:07 UTC] USER=www-data EUID=0 PID=3956046 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 05:47:07 UTC] USER=www-data EUID=0 PID=3956056 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:47:07 UTC] USER=www-data EUID=0 PID=3956068 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:47:07 UTC] USER=www-data EUID=0 PID=3956080 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:47:07 UTC] USER=www-data EUID=0 PID=3956089 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 05:47:07 UTC] USER=www-data EUID=0 PID=3956098 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
   ✅ Symlinked client-key.pem
[2026-02-05 05:47:07 UTC] USER=www-data EUID=0 PID=3956119 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 05:47:07 UTC] USER=www-data EUID=0 PID=3956130 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 05:47:07 UTC] USER=www-data EUID=0 PID=3956146 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: web-universe-main-dev
User: pgbouncer_admin
Node: pgbouncer
FQDN: db-web-universe-main-dev-postgresql-bouncer.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-web-universe-main-dev-postgresql-bouncer.fastorder.com -U pgbouncer_admin -d postgres

[0;32m[OK][0m   mTLS client certificate present: /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[0;34m[INFO][0m Creating symlinks to canonical certificates in /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend...
[2026-02-05 05:47:07 UTC] USER=www-data EUID=0 PID=3956173 ACTION=fsop ARGS=mkdir -p /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend
[2026-02-05 05:47:07 UTC] USER=www-data EUID=0 PID=3956182 ACTION=fsop ARGS=mkdir -p /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend
[2026-02-05 05:47:07 UTC] USER=www-data EUID=0 PID=3956191 ACTION=fsop ARGS=ln -sf /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-02-05 05:47:08 UTC] USER=www-data EUID=0 PID=3956203 ACTION=fsop ARGS=ln -sf /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[2026-02-05 05:47:08 UTC] USER=www-data EUID=0 PID=3956220 ACTION=fsop ARGS=ln -sf /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt
[0;34m[INFO][0m Creating coordinator CA symlink for PostgreSQL server verification...
[2026-02-05 05:47:08 UTC] USER=www-data EUID=0 PID=3956229 ACTION=fsop ARGS=ln -sf /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m Verifying canonical certificate permissions...
[2026-02-05 05:47:08 UTC] USER=www-data EUID=0 PID=3956238 ACTION=fsop ARGS=chmod 644 /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 05:47:08 UTC] USER=www-data EUID=0 PID=3956247 ACTION=fsop ARGS=chmod 640 /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 05:47:08 UTC] USER=www-data EUID=0 PID=3956256 ACTION=fsop ARGS=chmod 644 /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 05:47:08 UTC] USER=www-data EUID=0 PID=3956265 ACTION=fsop ARGS=chown root:www-data /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[0;32m[OK][0m   Backend certificate symlinks created in /etc/ssl
[0;32m[OK][0m   Coordinator CA symlink created for server verification
[0;34m[INFO][0m Creating symlinks in /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer for monitoring access...
[2026-02-05 05:47:08 UTC] USER=www-data EUID=0 PID=3956277 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 05:47:08 UTC] USER=www-data EUID=0 PID=3956316 ACTION=fsop ARGS=ln -sf /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
/bin/ln: '/home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key' and '/etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key' are the same file
[0;32m[OK][0m   Monitoring certificate symlinks created (or already exist)
[2026-02-05 05:47:08 UTC] USER=www-data EUID=0 PID=3956350 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
[2026-02-05 05:47:08 UTC] USER=www-data EUID=0 PID=3956362 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
[2026-02-05 05:47:08 UTC] USER=www-data EUID=0 PID=3956373 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 05:47:08 UTC] USER=www-data EUID=0 PID=3956383 ACTION=fsop ARGS=test -r /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m PgBouncer will use PostgreSQL coordinator CA: /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;32m[OK][0m   PostgreSQL coordinator at db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432 is reachable
[0;34m[INFO][0m Dumping SCRAM secrets from coordinator for PgBouncer auth_file …
[2026-02-05 05:47:09 UTC] USER=www-data EUID=0 PID=3956418 ACTION=fsop ARGS=cp /tmp/tmp.QixiW4XDME /etc/pgbouncer/web-universe-main-dev/userlist.txt
[2026-02-05 05:47:09 UTC] USER=www-data EUID=0 PID=3956436 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/userlist.txt
[2026-02-05 05:47:09 UTC] USER=www-data EUID=0 PID=3956455 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;32m[OK][0m   Auth file written: /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;34m[INFO][0m Retrieved password from vault for pgbouncer_admin
[0;34m[INFO][0m Ensuring PgBouncer admin role 'pgbouncer_admin' exists in Postgres (coordinator) …
[0;32m[OK][0m   Role pgbouncer_admin created/updated successfully
[0;34m[SECRETS][0m Setting credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/pgbouncer_admin
[0;32m✓ [SECRETS][0m Credentials updated in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/pgbouncer_admin
[0;34m[INFO][0m ✅ PgBouncer admin password stored in centralized secrets vault
[0;34m[INFO][0m Re-fetching SCRAM secrets after role creation to ensure pgbouncer_admin is included …
[2026-02-05 05:47:17 UTC] USER=www-data EUID=0 PID=3958374 ACTION=fsop ARGS=cp /tmp/tmp.fKS4hR87Lr /etc/pgbouncer/web-universe-main-dev/userlist.txt
[2026-02-05 05:47:17 UTC] USER=www-data EUID=0 PID=3958383 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/userlist.txt
[2026-02-05 05:47:17 UTC] USER=www-data EUID=0 PID=3958392 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;32m[OK][0m   Auth file updated with pgbouncer_admin SCRAM hash
[0;34m[INFO][0m Auth file contains [2026-02-05 05:47:17 UTC] USER=www-data EUID=0 PID=3958403 ACTION=passthru ARGS=bash -c wc -l < '/etc/pgbouncer/web-universe-main-dev/userlist.txt'
4 user(s)
[0;32m[OK][0m   Admin 'pgbouncer_admin' password generated and saved
[0;34m[INFO][0m Configuring PostgreSQL to prevent Citus metadata sync hangs...
ALTER ROLE
[0;32m[OK][0m   Disabled Citus metadata sync for pgbouncer_admin
[0;34m[INFO][0m Verifying application database fastorder_web_universe_main_dev_db exists...
[0;32m[OK][0m   ✓ Database fastorder_web_universe_main_dev_db exists
[0;34m[INFO][0m Granting permissions to pgbouncer_admin on fastorder_web_universe_main_dev_db...
GRANT
[0;32m[OK][0m   ✓ Granted CONNECT on fastorder_web_universe_main_dev_db to pgbouncer_admin
GRANT
[0;32m[OK][0m   ✓ Granted USAGE on schema public to pgbouncer_admin
GRANT
[0;32m[OK][0m   ✓ Granted SELECT on all tables to pgbouncer_admin
ALTER DATABASE
[0;32m[OK][0m   Set synchronous_commit=local for fastorder_web_universe_main_dev_db
[0;34m[INFO][0m Ensuring pg_hba.conf entry for pgbouncer_admin …
[0;34m[INFO][0m Adding pg_hba.conf entries for pgbouncer_admin with cert auth …
[2026-02-05 05:47:18 UTC] USER=unknown EUID=33 PID=3958455 ACTION=-u ARGS=postgres bash
ERROR: Invalid or unauthorized action: -u
[0;32m[OK][0m   pg_hba.conf updated and PostgreSQL configuration reloaded
[1;33m[WARN][0m pg_hba.conf entry may not have loaded correctly
[0;34m[INFO][0m Writing /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini …
[2026-02-05 05:47:19 UTC] USER=www-data EUID=0 PID=3958620 ACTION=fsop ARGS=cp /tmp/tmp.2waD4mwlCd /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
[2026-02-05 05:47:19 UTC] USER=www-data EUID=0 PID=3958629 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
[2026-02-05 05:47:19 UTC] USER=www-data EUID=0 PID=3958638 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
[2026-02-05 05:47:19 UTC] USER=www-data EUID=0 PID=3958647 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbouncer/web-universe-main-dev /run/pgbouncer/web-universe-main-dev /var/log/pgbouncer/web-universe-main-dev
[2026-02-05 05:47:19 UTC] USER=www-data EUID=0 PID=3958656 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;32m[OK][0m   pgbouncer.ini ready
[0;34m[INFO][0m Verifying TLS settings in pgbouncer.ini:
[2026-02-05 05:47:19 UTC] USER=www-data EUID=0 PID=3958667 ACTION=fsop ARGS=grep -E (client_tls_sslmode|server_tls) /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
[0;34m[INFO][0m Verifying PgBouncer server certificate files:
[2026-02-05 05:47:19 UTC] USER=www-data EUID=0 PID=3958676 ACTION=fsop ARGS=test -r /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[0;32m[OK][0m   Server cert readable by postgres: /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-02-05 05:47:19 UTC] USER=www-data EUID=0 PID=3958686 ACTION=fsop ARGS=test -r /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;32m[OK][0m   Server key readable by postgres: /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;34m[INFO][0m Verifying coordinator CA certificate:
[2026-02-05 05:47:19 UTC] USER=www-data EUID=0 PID=3958697 ACTION=fsop ARGS=test -r /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;32m[OK][0m   Coordinator CA readable by postgres: /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m Preflight: stopping any conflicting PgBouncer on 6432 …
[2026-02-05 05:47:19 UTC] USER=www-data EUID=0 PID=3958708 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer.service
[2026-02-05 05:47:20 UTC] USER=www-data EUID=0 PID=3958717 ACTION=passthru ARGS=systemctl stop pgbouncer@web-universe-main-dev.service
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json?all=1": dial unix /var/run/docker.sock: connect: permission denied
[2026-02-05 05:47:22 UTC] USER=www-data EUID=0 PID=3958828 ACTION=passthru ARGS=systemctl daemon-reload
[0;32m[OK][0m   systemd unit installed: pgbouncer@web-universe-main-dev.service
[0;34m[INFO][0m Running pre-flight IP conflict check for 10.100.1.244:6432 …
[1;33m[WARN][0m IP conflict checker not found at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/check-ip-conflicts.sh
[1;33m[WARN][0m Skipping pre-flight check - conflicts may occur
[0;34m[INFO][0m Starting PgBouncer (web-universe-main-dev) …
[2026-02-05 05:47:23 UTC] USER=www-data EUID=0 PID=3958972 ACTION=passthru ARGS=systemctl restart pgbouncer@web-universe-main-dev.service
[2026-02-05 05:47:23 UTC] USER=www-data EUID=0 PID=3958984 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer@web-universe-main-dev.service
[0;32m[OK][0m   Service ACTIVE
[0;34m[INFO][0m Verifying auth_file before probing …
[0;34m[INFO][0m Auth file contains 4 user(s)
[1;33m[WARN][0m Auth file does NOT contain pgbouncer_admin entry - authentication will fail
[0;34m[INFO][0m Probing admin console via SSL (psql to database 'pgbouncer') …
[0;34m[INFO][0m Retrieved password from vault for admin console probe
[1;33m[WARN][0m Admin console probe failed (see error below)
psql: error: connection to server at "10.100.1.244", port 6432 failed: root certificate file "/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
[1;33m[WARN][0m Troubleshooting:
[1;33m[WARN][0m   1. Check auth_file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/web-universe-main-dev/userlist.txt
[1;33m[WARN][0m   2. Test with: PGPASSWORD='WO0D2C0d7ZbIdk65D10y9TaD' psql -h 10.100.1.244 -p 6432 -U pgbouncer_admin -d pgbouncer
[1;33m[WARN][0m   3. Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@web-universe-main-dev.service -n 50

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m   Running Comprehensive PgBouncer Verification Tests
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[0;34m[INFO][0m Password extracted: WO0D2C0d7Z... (using postgres user certificates)

[0;34m[INFO][0m Test 1/7: Admin Console - SHOW POOLS
[1;33m[WARN][0m ✗ SHOW POOLS: FAILED
[1;33m[WARN][0m Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@web-universe-main-dev.service -n 50

[0;34m[INFO][0m Test 2/7: Admin Console - SHOW VERSION
[1;33m[WARN][0m ✗ SHOW VERSION: FAILED

[0;34m[INFO][0m Test 3/7: Admin Console - SHOW STATS
[1;33m[WARN][0m ✗ SHOW STATS: FAILED

[0;34m[INFO][0m Test 4/7: Admin Console - SHOW DATABASES
[1;33m[WARN][0m ✗ SHOW DATABASES: FAILED

[0;34m[INFO][0m Test 5/7: Admin Console - SHOW CONFIG
[1;33m[WARN][0m ✗ SHOW CONFIG: FAILED
psql   "host=db-web-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_web_universe_main_dev_db user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD    connect_timeout=5 sslmode=verify-full    sslrootcert=/home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt    sslcert=/home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt    sslkey=/home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key"   --no-psqlrc -Atc 'SELECT version();'

[0;34m[INFO][0m Test 6/7: Application Database - SELECT version()
[1;33m[WARN][0m ✗ Application database query: FAILED (timeout or connection issue)
[1;33m[WARN][0m    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[0;34m[INFO][0m Test 7/8: Application Database - Connection Details
[1;33m[WARN][0m ✗ Connection details: FAILED (timeout or connection issue)
[1;33m[WARN][0m    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[0;34m[INFO][0m Test 8/8: End-to-End Application Routing - Pool Verification
[0;34m[INFO][0m   Running actual queries through PgBouncer to verify routing and pooling...
[1;33m[WARN][0m ✗ End-to-end routing verification: FAILED - All 3 queries failed
[1;33m[WARN][0m    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[1;33m[WARN][0m    Otherwise check if database fastorder_web_universe_main_dev_db exists and user pgbouncer_admin has permissions

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m   Verification Complete - Tests 1-5 PASSED (Admin console verified)
[1;33m[WARN][0m   Tests 6-8 FAILED - Application database not accessible
[1;33m[WARN][0m   This is expected if Citus is not set up yet
[1;33m[WARN][0m   Run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[0;32m[OK][0m   PgBouncer is up for web-universe-main-dev

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Connection Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Password stored in: AWS Secrets Manager (fastorder/db/web/ksa/main/dev/postgresqlweb/universe/main/dev/coordinator-pgbouncer_admin)
Current password: WO0D2C0d7ZbIdk65D10y9TaD

1. Admin Console (using IP address to avoid DNS/SSL issues):
   psql "host=10.100.1.244 port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD sslmode=verify-full sslrootcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

2. Admin Console (using hostname):
   psql "host=db-web-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD sslmode=verify-full sslrootcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

3. Application Database:
   psql "host=db-web-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_web_universe_main_dev_db sslkey=/etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD sslmode=verify-full sslrootcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

4. Using .pgpass file:
   echo "db-web-universe-main-dev-postgresql-bouncer.fastorder.com:6432:*:pgbouncer_admin:WO0D2C0d7ZbIdk65D10y9TaD" >> ~/.pgpass
   chmod 600 ~/.pgpass
   psql -h db-web-universe-main-dev-postgresql-bouncer.fastorder.com -p 6432 -U pgbouncer_admin -d fastorder_web_universe_main_dev_db

5. Retrieve password from vault:
   source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
   PGPASSWORD="$(get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password')" \
     psql -h 10.100.1.244 -p 6432 -U pgbouncer_admin -d pgbouncer -c "SHOW POOLS;"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Architecture
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  • Default db 'fastorder_web_universe_main_dev_db' → Citus coordinator (db-web-universe-main-dev-postgresql-coordinator.fastorder.com)
  • Worker access: 'fastorder_web_universe_main_dev_db_worker_1', 'fastorder_web_universe_main_dev_db_worker_2', … (if exist)
  • Client TLS: require (password auth) / verify-full (mTLS with certs)
  • Server TLS: verify-full (PgBouncer validates PostgreSQL certs)
  • Auth: SCRAM-SHA-256 via /etc/pgbouncer/web-universe-main-dev/userlist.txt
  • Pool mode: transaction (stateless connections)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Management
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Service Status:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer@web-universe-main-dev.service
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@web-universe-main-dev.service

Logs:
  command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@web-universe-main-dev.service -f
  /usr/local/bin/fastorder-provisioning-wrapper.sh tail -f /var/log/pgbouncer/web-universe-main-dev/pgbouncer.log

Reload Config:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@web-universe-main-dev.service

Restart:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@web-universe-main-dev.service

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Files
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Config:        /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
Auth file:     /etc/pgbouncer/web-universe-main-dev/userlist.txt
Server cert:   /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
Server key:    /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
CA cert:       /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt
PG CA:         /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
Logs:          /var/log/pgbouncer/web-universe-main-dev/pgbouncer.log

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Troubleshooting
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


If "SASL authentication failed":
  1. Check auth file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/web-universe-main-dev/userlist.txt
  2. Verify pgbouncer_admin is present with SCRAM hash
  3. Get password from vault:
     source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
     get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password'
  4. Reload PgBouncer: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@web-universe-main-dev.service

If "no pg_hba.conf entry":
  1. Check pg_hba.conf on coordinator
  2. Add rule: hostssl all pgbouncer_admin 10.100.1.244/32 cert clientcert=verify-full
  3. Reload PostgreSQL

To add users to PgBouncer:
  1. Create user in PostgreSQL with password
  2. Re-run SCRAM dump:
     psql "host=db-web-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 dbname=postgres user=postgres \
       sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt \
       sslcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt sslkey=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key" \
       -Atc "SELECT '\"' || rolname || '\" \"' || rolpassword || '\"' \
             FROM pg_authid WHERE rolpassword LIKE 'SCRAM-SHA-256%' \
             AND rolcanlogin ORDER BY rolname;" | command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop tee /etc/pgbouncer/web-universe-main-dev/userlist.txt
  3. Reload: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@web-universe-main-dev.service

[0;34m[INFO][0m Registering PgBouncer node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        web-universe-main-dev-pgbouncer
[INFO]   Identifier Parent: postgresql
[INFO]   IP:                10.100.1.244
[INFO]   Port:              6432
[INFO]   FQDN:              db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: fdc70c5f-615d-432f-8161-a7acd56ea9ed
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[0;32m[OK][0m   PgBouncer node registered to observability API
[0;32m✓[0m ✅ PgBouncer setup completed

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 03-citus-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m CITUS DISTRIBUTED CLUSTER SETUP
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Phase 1: Installing Citus extension on workers...
[0;34m[INFO][0m Phase 2: Setting up coordinator and registering workers...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[0;34m[INFO][0m 📦 PHASE 1: Installing Citus extension on 1 worker(s)...

[0;34m[INFO][0m → Worker 1/1: Installing Citus on worker-01...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS CLUSTER SETUP
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════

[0;34m[INFO][0m 🔧 Setting up Citus Worker...
[0;34m[INFO][0m Temporarily disabling synchronous replication for extension installation...
t
[0;34m[INFO][0m Installing Citus extension on worker...
[0;32m[OK][0m   Citus extension installed on worker
[0;34m[INFO][0m Restoring synchronous replication settings...
t
[0;34m[INFO][0m Worker Citus extension installed - registration will happen when coordinator setup runs

[0;32m[OK][0m   Citus setup complete for worker-01
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m✓[0m   ✅ Citus extension installed on worker-01

[0;32m✓[0m ✅ Phase 1 Complete: All 1 workers have Citus extension installed

[0;34m[INFO][0m 🔧 PHASE 2: Setting up Citus coordinator and registering workers...

[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS CLUSTER SETUP
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════

[0;34m[INFO][0m 🔧 Setting up Citus Coordinator...

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m DIAGNOSTIC: Configuration Variables
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m PG_WORKERS_NUM: 1
[0;34m[INFO][0m ENV_ID: web-universe-main-dev
[0;34m[INFO][0m DOMAIN: fastorder.com
[0;34m[INFO][0m PORT: 5432
[0;34m[INFO][0m SOCKET_DIR: /var/run/postgresql-web-universe-main-dev-coordinator
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[0;34m[INFO][0m Ensuring postgres client certificates exist for coordinator...
[0;32m[OK][0m   Postgres client certificates already exist for coordinator
[0;34m[INFO][0m Adding citus_cert_map to coordinator pg_ident.conf...
[0;32m[OK][0m   pg_ident.conf updated for coordinator
[0;34m[INFO][0m Installing Citus extension on coordinator...
[0;32m[OK][0m   Citus extension installed on coordinator (postgres database)
[0;34m[INFO][0m Installing Citus extension on application database: fastorder_web_universe_main_dev_db...
[0;32m[OK][0m   Citus extension installed on application database: fastorder_web_universe_main_dev_db
[0;34m[INFO][0m Configuring Citus SSL connection parameters...
[2026-02-05 05:47:37 UTC] USER=www-data EUID=0 PID=3960604 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m   ✅ Citus SSL connection parameters configured: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[1;33m[WARN][0m Node not identified as coordinator, initializing...
[0;34m[INFO][0m Checking coordinator configuration...
[0;34m[INFO][0m Persisting citus.local_hostname to postgresql.conf...
[2026-02-05 05:47:40 UTC] USER=www-data EUID=0 PID=3961395 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/web-universe-main-dev/coordinator/postgresql.conf
[2026-02-05 05:47:40 UTC] USER=www-data EUID=0 PID=3961443 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m   ✅ citus.local_hostname persisted to config and reloaded
[0;34m[INFO][0m Configuring coordinator hostname in postgres database: db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432

[0;32m[OK][0m   ✅ Coordinator hostname set to db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432 in postgres database
[0;34m[INFO][0m Checking coordinator configuration in application database: fastorder_web_universe_main_dev_db...
[0;34m[INFO][0m Configuring coordinator hostname in application database: db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432
[0;32m[OK][0m   ✅ Coordinator hostname set to db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432 in application database
[0;34m[INFO][0m Validating coordinator configuration before worker registration...
[0;32m[OK][0m   ✅ Coordinator hostname validated: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[0;32m[OK][0m   ✅ citus_tables view is accessible
[0;34m[INFO][0m Checking coordinator self-registration...
[0;32m[OK][0m   ✅ Coordinator is already self-registered
[0;34m[INFO][0m Configuring coordinator shard placement policy...
[0;32m[OK][0m   ✅ Coordinator already configured in postgres database (shouldhaveshards = false)
[0;32m[OK][0m   ✅ Coordinator already configured in application database (shouldhaveshards = false)
[0;34m[INFO][0m Registering 1 worker(s) to Citus cluster...

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m PRE-FLIGHT: Checking worker availability...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Checking worker worker-01...
[0;34m[INFO][0m   FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[0;32m[OK][0m   ✅ Worker worker-01 is reachable via SSL
[0;32m[OK][0m   All workers are reachable - proceeding with registration

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Adding Citus worker: db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Adding citus_cert_map to worker-01 pg_ident.conf...
[0;32m[OK][0m   pg_ident.conf updated for worker-01
[0;34m[INFO][0m Configuring worker worker-01 HBA for coordinator (10.100.1.119) access...
[0;32m[OK][0m   Worker worker-01 HBA configured for coordinator (10.100.1.119)
[0;34m[INFO][0m Adding replication rules for 3 standby(s)...
[0;32m[OK][0m   Replication rules added for worker-01
[0;34m[INFO][0m Reloading worker worker-01 to apply HBA changes...
[2026-02-05 05:47:43 UTC] USER=www-data EUID=0 PID=3961784 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-worker-01.service
[0;34m[INFO][0m Configuring coordinator HBA for worker worker-01 (10.100.1.243) access...
[0;32m[OK][0m   Coordinator HBA configured for worker worker-01 (10.100.1.243)
[0;34m[INFO][0m Reloading coordinator to apply HBA changes...
[2026-02-05 05:47:43 UTC] USER=www-data EUID=0 PID=3961816 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
[0;34m[INFO][0m Ensuring postgres client certificates exist for worker-01...
[0;32m[OK][0m   Postgres client certificates already exist for worker-01
[0;34m[INFO][0m Configuring citus.node_conninfo on worker-01...
[2026-02-05 05:47:44 UTC] USER=www-data EUID=0 PID=3961832 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-worker-01.service
[0;32m[OK][0m   citus.node_conninfo configured on worker-01
[0;34m[INFO][0m Temporarily relaxing sync-rep on worker worker-01...
t
[0;32m[OK][0m   Worker worker-01 sync-rep relaxed (was: sync_commit=on)
[0;34m[INFO][0m Ensuring Citus extension on worker databases...
CREATE EXTENSION
CREATE EXTENSION
[0;34m[INFO][0m Running citus_add_node with 180s timeout...
2
[0;34m[INFO][0m Restoring worker worker-01 sync-rep settings...
t
[0;32m[OK][0m   Worker worker-01 sync-rep restored
[0;32m[OK][0m   ✅ Worker db-web-universe-main-dev-postgresql-worker-01.fastorder.com successfully added to Citus cluster
[0;34m[INFO][0m    Node ID: 2
[0;34m[INFO][0m    Registered in: postgres, fastorder_web_universe_main_dev_db
[0;32m[OK][0m   Worker worker-01 registration successful
[0;34m[INFO][0m Configuring worker worker-01 shard placement policy...
[0;32m[OK][0m   ✅ Worker worker-01 configured to hold shards in all databases


[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m POST-REGISTRATION: Verifying cluster state...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Expected workers: 1
[0;34m[INFO][0m Registered workers: 1
[0;32m[OK][0m   ✅ All 1 workers successfully registered!

[0;34m[INFO][0m Citus cluster configuration:
db-web-universe-main-dev-postgresql-coordinator.fastorder.com  5432  0  t  primary  f
db-web-universe-main-dev-postgresql-worker-01.fastorder.com    5432  1  t  primary  t

[0;34m[INFO][0m Note: groupid=0 is the coordinator, groupid>0 are workers
[0;34m[INFO][0m       shouldhaveshards: false=query router only, true=holds data shards

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m FINAL VALIDATION: Verifying configuration persistence...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 05:47:47 UTC] USER=www-data EUID=0 PID=3962075 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/web-universe-main-dev/coordinator/postgresql.conf
[0;32m[OK][0m   ✅ citus.local_hostname persisted in postgresql.conf
[0;32m[OK][0m   ✅ All 1 worker(s) successfully registered and verified

[0;32m[OK][0m   ✅ All validation checks passed
[0;32m[OK][0m   Citus coordinator setup complete

[0;32m[OK][0m   Citus setup complete for coordinator
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════

[0;32m✓[0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✓[0m ✅ CITUS CLUSTER SETUP COMPLETED SUCCESSFULLY
[0;32m✓[0m    Coordinator: Ready and accepting connections
[0;32m✓[0m    Workers registered: 1
[0;32m✓[0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 05-backup-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up coordinator backup...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for web-universe-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-02-05 05:47:49 UTC] USER=www-data EUID=0 PID=3962202 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 05:47:49 UTC] USER=www-data EUID=0 PID=3962211 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/web-universe-main-dev
[2026-02-05 05:47:49 UTC] USER=www-data EUID=0 PID=3962226 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-02-05 05:47:49 UTC] USER=www-data EUID=0 PID=3962237 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-02-05 05:47:49 UTC] USER=www-data EUID=0 PID=3962257 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-02-05 05:47:49 UTC] USER=www-data EUID=0 PID=3962269 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-02-05 05:47:51 UTC] USER=www-data EUID=0 PID=3962384 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-02-05 05:47:51 UTC] USER=www-data EUID=0 PID=3962398 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-02-05 05:47:52 UTC] USER=www-data EUID=0 PID=3962410 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-02-05 05:47:52 UTC] USER=www-data EUID=0 PID=3962422 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 05:47:52 UTC] USER=www-data EUID=0 PID=3962431 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-web-universe-main-dev
[2026-02-05 05:47:52 UTC] USER=www-data EUID=0 PID=3962452 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-02-05 05:47:52 UTC] USER=www-data EUID=0 PID=3962472 ACTION=fsop ARGS=find /data/postgresql/17/web-universe-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-02-05 05:47:52 UTC] USER=www-data EUID=0 PID=3962481 ACTION=fsop ARGS=chown -R postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-02-05 05:47:52 UTC] USER=www-data EUID=0 PID=3962490 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-02-05 05:47:52 UTC] USER=www-data EUID=0 PID=3962500 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-02-05 05:47:52 UTC] USER=www-data EUID=0 PID=3962514 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-02-05 05:47:52 UTC] USER=www-data EUID=0 PID=3962524 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/coordinator/PG_VERSION
[2026-02-05 05:47:52 UTC] USER=www-data EUID=0 PID=3962542 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza web-universe-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-02-05 05:47:52 UTC] USER=www-data EUID=0 PID=3962603 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-02-05 05:47:55 UTC] USER=www-data EUID=0 PID=3962731 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-02-05 05:47:59 UTC] USER=www-data EUID=0 PID=3963268 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator --log-level-console=info check
2026-02-05 05:47:59.810 P00   INFO: check command begin 2.56.0: --exec-id=3963277-5c5da3a4 --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 05:47:59.838 P00   INFO: check repo1 configuration (primary)
2026-02-05 05:47:59.850 P00  ERROR: [028]: backup and archive info files exist but do not match the database
                                    HINT: is this the correct stanza?
                                    HINT: did an error occur during stanza-upgrade?
2026-02-05 05:47:59.851 P00   INFO: check command end: aborted with exception [028]
[WARN] ⚠️  Stanza verification failed - this may be normal if WAL archiving hasn't started yet
[WARN]    The backup system is configured and will work once WAL segments are generated

[INFO] 8️⃣ Creating backup automation scripts...
[2026-02-05 05:47:59 UTC] USER=www-data EUID=0 PID=3963290 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 05:47:59 UTC] USER=www-data EUID=0 PID=3963299 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 05:48:00 UTC] USER=www-data EUID=0 PID=3963319 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[2026-02-05 05:48:00 UTC] USER=www-data EUID=0 PID=3963335 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-02-05 05:48:00 UTC] USER=www-data EUID=0 PID=3963359 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-web-universe-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-02-05 05:48:00 UTC] USER=www-data EUID=0 PID=3963379 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 05:48:00 UTC] USER=www-data EUID=0 PID=3963388 ACTION=fsop ARGS=sed -i s|__ENV_ID__|web-universe-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 05:48:00 UTC] USER=www-data EUID=0 PID=3963397 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/data/postgresql/17/web-universe-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 05:48:00 UTC] USER=www-data EUID=0 PID=3963406 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 05:48:00 UTC] USER=www-data EUID=0 PID=3963425 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-02-05 05:48:00.894 P00   INFO: start command begin 2.56.0: --exec-id=3963487-f3a481cc --log-level-console=info --log-level-file=debug --stanza=web-universe-main-dev-coordinator
2026-02-05 05:48:00.894 P00   WARN: stop file does not exist for stanza web-universe-main-dev-coordinator
2026-02-05 05:48:00.894 P00   INFO: start command end: completed successfully (15ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-02-05 05:48:00.987 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=3963525-9dc92faf --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 05:48:00.988 P00   INFO: stanza-upgrade for stanza 'web-universe-main-dev-coordinator' on repo1
2026-02-05 05:48:01.012 P00   INFO: stanza-upgrade command end: completed successfully (30ms)
[INFO] This may take a few minutes depending on database size...
[2026-02-05 05:48:01 UTC] USER=www-data EUID=0 PID=3963548 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260205-054801.log
[2026-02-05 05:48:01 UTC] USER=www-data EUID=0 PID=3963568 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260205-054801.log
[2026-02-05 05:48:01 UTC] USER=www-data EUID=0 PID=3963593 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260205-054801.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-02-05 05:48:10 UTC] USER=www-data EUID=0 PID=3964519 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-3962171.log /var/log/pgbackrest/initial-backup-20260205-054801.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260205-054801.log
   2026-02-05 05:48:10.466 P00   INFO: repo1: remove expired backup 20260205-045754F
   2026-02-05 05:48:10.548 P00   INFO: repo1: 17-1 remove archive, start = 000000010000000000000004, stop = 000000010000000000000006
   2026-02-05 05:48:10.549 P00   INFO: repo1: 17-2 no archive to remove
   2026-02-05 05:48:10.553 P00   INFO: repo1: 17-3 remove archive, start = 000000010000000000000002, stop = 000000010000000000000002
   2026-02-05 05:48:10.553 P00   INFO: expire command end: completed successfully (92ms)

[INFO] Current backups:
stanza: web-universe-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000007/00000001000000000000000A

        full backup: 20260205-045818F
            timestamp start/stop: 2026-02-05 04:58:18+00 / 2026-02-05 04:58:24+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.7MB, database backup size: 37.7MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

    db (prior)
        wal archive min/max (17): 000000010000000000000003/00000001000000000000000C

        full backup: 20260205-051721F
            timestamp start/stop: 2026-02-05 05:17:21+00 / 2026-02-05 05:17:34+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 33.5MB, database backup size: 33.5MB
            repo1: backup set size: 5.4MB, backup size: 5.4MB

        full backup: 20260205-051746F
            timestamp start/stop: 2026-02-05 05:17:46+00 / 2026-02-05 05:17:48+00
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 33.5MB, database backup size: 33.5MB
            repo1: backup set size: 5.4MB, backup size: 5.4MB

    db (current)
        wal archive min/max (17): none present

        full backup: 20260205-054801F
            timestamp start/stop: 2026-02-05 05:48:01+00 / 2026-02-05 05:48:10+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 33.5MB, database backup size: 33.5MB
            repo1: backup set size: 5.4MB, backup size: 5.4MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         web-universe-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/web-universe-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up worker backups for 1 worker(s)...
[0;34m[INFO][0m Setting up backup for: worker-01
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for web-universe-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-02-05 05:48:11 UTC] USER=www-data EUID=0 PID=3964584 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 05:48:11 UTC] USER=www-data EUID=0 PID=3964593 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/web-universe-main-dev
[2026-02-05 05:48:11 UTC] USER=www-data EUID=0 PID=3964602 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-02-05 05:48:11 UTC] USER=www-data EUID=0 PID=3964611 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-02-05 05:48:11 UTC] USER=www-data EUID=0 PID=3964622 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-02-05 05:48:11 UTC] USER=www-data EUID=0 PID=3964631 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-02-05 05:48:13 UTC] USER=www-data EUID=0 PID=3964714 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-02-05 05:48:13 UTC] USER=www-data EUID=0 PID=3964737 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-02-05 05:48:13 UTC] USER=www-data EUID=0 PID=3964765 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 05:48:13 UTC] USER=www-data EUID=0 PID=3964782 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-web-universe-main-dev
[2026-02-05 05:48:14 UTC] USER=www-data EUID=0 PID=3964831 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-02-05 05:48:14 UTC] USER=www-data EUID=0 PID=3964843 ACTION=fsop ARGS=find /data/postgresql/17/web-universe-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-02-05 05:48:14 UTC] USER=www-data EUID=0 PID=3964881 ACTION=fsop ARGS=chown -R postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-02-05 05:48:14 UTC] USER=www-data EUID=0 PID=3964911 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-02-05 05:48:14 UTC] USER=www-data EUID=0 PID=3964927 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-02-05 05:48:14 UTC] USER=www-data EUID=0 PID=3964950 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-02-05 05:48:14 UTC] USER=www-data EUID=0 PID=3964978 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/coordinator/PG_VERSION
[2026-02-05 05:48:15 UTC] USER=www-data EUID=0 PID=3964989 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza web-universe-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-02-05 05:48:16 UTC] USER=www-data EUID=0 PID=3965120 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-02-05 05:48:19 UTC] USER=www-data EUID=0 PID=3965491 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 05:48:23 UTC] USER=www-data EUID=0 PID=3965975 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-02-05 05:48:23 UTC] USER=www-data EUID=0 PID=3966008 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator --log-level-console=info check
2026-02-05 05:48:23.499 P00   INFO: check command begin 2.56.0: --exec-id=3966015-f4574d27 --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 05:48:23.523 P00   INFO: check repo1 configuration (primary)
2026-02-05 05:48:23.585 P00   INFO: check repo1 archive for WAL (primary)
2026-02-05 05:48:24.186 P00   INFO: WAL segment 000000010000000000000005 successfully archived to '/var/lib/pgbackrest/backup/web-universe-main-dev/archive/web-universe-main-dev-coordinator/17-3/0000000100000000/000000010000000000000005-7a32accdd94768bccb42dcb080f3c1d105179278.lz4' on repo1
2026-02-05 05:48:24.186 P00   INFO: check command end: completed successfully (691ms)
[INFO] ✅ Stanza verification passed

[INFO] 8️⃣ Creating backup automation scripts...
[2026-02-05 05:48:24 UTC] USER=www-data EUID=0 PID=3966074 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 05:48:24 UTC] USER=www-data EUID=0 PID=3966093 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 05:48:24 UTC] USER=www-data EUID=0 PID=3966131 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[2026-02-05 05:48:24 UTC] USER=www-data EUID=0 PID=3966150 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-02-05 05:48:24 UTC] USER=www-data EUID=0 PID=3966201 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-web-universe-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-02-05 05:48:24 UTC] USER=www-data EUID=0 PID=3966273 ACTION=fsop ARGS=sed -i s|__ENV_ID__|web-universe-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 05:48:24 UTC] USER=www-data EUID=0 PID=3966293 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/data/postgresql/17/web-universe-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 05:48:24 UTC] USER=www-data EUID=0 PID=3966319 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 05:48:24 UTC] USER=www-data EUID=0 PID=3966332 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-02-05 05:48:25.110 P00   INFO: start command begin 2.56.0: --exec-id=3966360-618b0918 --log-level-console=info --log-level-file=debug --stanza=web-universe-main-dev-coordinator
2026-02-05 05:48:25.110 P00   WARN: stop file does not exist for stanza web-universe-main-dev-coordinator
2026-02-05 05:48:25.110 P00   INFO: start command end: completed successfully (9ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-02-05 05:48:25.406 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=3966444-dc34a33e --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 05:48:25.407 P00   INFO: stanza-upgrade for stanza 'web-universe-main-dev-coordinator' on repo1
2026-02-05 05:48:25.409 P00   INFO: stanza 'web-universe-main-dev-coordinator' on repo1 is already up to date
2026-02-05 05:48:25.409 P00   INFO: stanza-upgrade command end: completed successfully (13ms)
[INFO] This may take a few minutes depending on database size...
[2026-02-05 05:48:25 UTC] USER=www-data EUID=0 PID=3966461 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260205-054825.log
[2026-02-05 05:48:25 UTC] USER=www-data EUID=0 PID=3966478 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260205-054825.log
[2026-02-05 05:48:25 UTC] USER=www-data EUID=0 PID=3966488 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260205-054825.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-02-05 05:48:34 UTC] USER=www-data EUID=0 PID=3966992 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-3964555.log /var/log/pgbackrest/initial-backup-20260205-054825.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260205-054825.log
   2026-02-05 05:48:33.976 P00   INFO: repo1: remove expired backup 20260205-045818F
   2026-02-05 05:48:34.013 P00   INFO: repo1: remove archive path /var/lib/pgbackrest/backup/web-universe-main-dev/archive/web-universe-main-dev-coordinator/17-1
   2026-02-05 05:48:34.013 P00   INFO: repo1: 17-2 no archive to remove
   2026-02-05 05:48:34.014 P00   INFO: repo1: 17-3 no archive to remove
   2026-02-05 05:48:34.014 P00   INFO: expire command end: completed successfully (43ms)

[INFO] Current backups:
stanza: web-universe-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000003/00000001000000000000000C

        full backup: 20260205-051721F
            timestamp start/stop: 2026-02-05 05:17:21+00 / 2026-02-05 05:17:34+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 33.5MB, database backup size: 33.5MB
            repo1: backup set size: 5.4MB, backup size: 5.4MB

        full backup: 20260205-051746F
            timestamp start/stop: 2026-02-05 05:17:46+00 / 2026-02-05 05:17:48+00
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 33.5MB, database backup size: 33.5MB
            repo1: backup set size: 5.4MB, backup size: 5.4MB

    db (current)
        wal archive min/max (17): 000000010000000000000003/000000010000000000000005

        full backup: 20260205-054801F
            timestamp start/stop: 2026-02-05 05:48:01+00 / 2026-02-05 05:48:10+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 33.5MB, database backup size: 33.5MB
            repo1: backup set size: 5.4MB, backup size: 5.4MB

        full backup: 20260205-054825F
            timestamp start/stop: 2026-02-05 05:48:25+00 / 2026-02-05 05:48:33+00
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 33.5MB, database backup size: 33.5MB
            repo1: backup set size: 5.4MB, backup size: 5.4MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         web-universe-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/web-universe-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✓[0m ✅ Backup setup completed for coordinator and all workers

[0;34m[INFO][0m Skipping 06-distribute-tables-canary.sh (test script - set RUN_TESTS=true to enable)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 07-distribute-tables.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 05:48:35 UTC] USER=unknown EUID=33 PID=3967057 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 05:48:35 UTC] USER=unknown EUID=33 PID=3967064 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 05:48:35 UTC] USER=unknown EUID=33 PID=3967073 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 05:48:35 UTC] USER=unknown EUID=33 PID=3967080 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS TABLE DISTRIBUTION
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════

[0;34m[INFO][0m 🔐 Secure connection established
[0;34m[INFO][0m    Host: db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432
[0;34m[INFO][0m    Database: fastorder_web_universe_main_dev_db
[0;34m[INFO][0m    SSL: verify-full (TLS 1.2+)
[0;34m[INFO][0m    Timeouts: statement=120s, idle_tx=300s

[0;34m[INFO][0m 🔍 Running preflight checks...
[0;34m[INFO][0m Testing database connectivity...
[0;32m[OK][0m   ✅ Database connection successful
[0;32m[OK][0m   ✅ Connected to correct database: fastorder_web_universe_main_dev_db
[0;34m[INFO][0m Checking Citus extension in database fastorder_web_universe_main_dev_db...
[0;32m[OK][0m   Citus version: 13.2-1
[0;34m[INFO][0m Checking worker registration...
[0;32m[OK][0m   Registered workers: 1
[0;34m[INFO][0m Worker nodes:
[0;34m[INFO][0m                             nodename                           | nodeport | isactive | noderole 
[0;34m[INFO][0m   -------------------------------------------------------------+----------+----------+----------
[0;34m[INFO][0m    db-web-universe-main-dev-postgresql-worker-01.fastorder.com |     5432 | t        | primary
[0;34m[INFO][0m   (1 row)
[0;34m[INFO][0m   

[0;34m[INFO][0m 📊 Starting table distribution...

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Distributing: auth.login_account
[0;34m[INFO][0m Description: User authentication table - distributed by region for tenant isolation
[0;34m[INFO][0m Shard key: region_hint
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ⏭️  Table does not exist, skipping

[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m[OK][0m   ✅ All tables distributed successfully!
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════

[0;34m[INFO][0m 📊 Citus Cluster Summary:

[0;34m[INFO][0m Distributed tables:
[0;34m[INFO][0m    table | type | shard_key | shards | size 
[0;34m[INFO][0m   -------+------+-----------+--------+------
[0;34m[INFO][0m   (0 rows)
[0;34m[INFO][0m   

[0;34m[INFO][0m Worker capacity:
[0;34m[INFO][0m    worker | total_shards | total_size 
[0;34m[INFO][0m   --------+--------------+------------
[0;34m[INFO][0m   (0 rows)
[0;34m[INFO][0m   

[0;32m[OK][0m   Citus table distribution complete

[0;34m[INFO][0m Skipping 08-distribute-tables-rollback.sh (rollback script - run manually only)
[0;34m[INFO][0m Skipping 09-distribute-tables-test.sh (test script - set RUN_TESTS=true to enable)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 10-setup-cdc.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m CDC PIPELINE SETUP (Debezium + Elasticsearch Sink)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Log file: /var/log/fastorder/cdc/10-setup-cdc-*.log

[0;34m[INFO][0m Running CDC setup for identifier: coordinator
[2026-02-05 05:48:39] ==========================================
[2026-02-05 05:48:39] CDC SETUP SCRIPT STARTED
[2026-02-05 05:48:39] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260205_054839.log
[2026-02-05 05:48:39] ==========================================
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 05:48:40] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 05:48:40]   CDC Pipeline Setup (Debezium + ES Sink)
[2026-02-05 05:48:40] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 05:48:40]   Environment: web-universe-main-dev
[2026-02-05 05:48:40]   Identifier:  coordinator
[2026-02-05 05:48:40]   Service:     web
[2026-02-05 05:48:40] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 05:48:40] 📂 CDC_BASE_DIR exists: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc
[2026-02-05 05:48:40] Looking for service folder: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web
[2026-02-05 05:48:40] 
[2026-02-05 05:48:40] 📂 Found CDC configuration for service: web
[2026-02-05 05:48:40] Scanning for subservice directories in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web
[2026-02-05 05:48:40] Found subservice: config, checking for steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps
[2026-02-05 05:48:40] 
[2026-02-05 05:48:40] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 05:48:40]   Setting up CDC for: web/config
[2026-02-05 05:48:40] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 05:48:40] Found 3 step script(s) in /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps
[2026-02-05 05:48:40] 
[2026-02-05 05:48:40] 🔧 Running: 01-setup-config-cdc.sh
[2026-02-05 05:48:40]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps/01-setup-config-cdc.sh
[2026-02-05 05:48:40]    Executing directly (script is executable)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Config CDC Setup - Automatic Role Detection
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Service:     web
[INFO]   Zone:        universe
[INFO]   Branch:      main
[INFO]   Environment: dev
[INFO]   Identifier:  coordinator
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service 'web' requires config.* schema
[INFO] CDC Role for web in zone universe: master
[INFO] ════════════════════════════════════════════════════════════
[INFO]   CONTROL PLANE MASTER (zone=universe)
[INFO]   Setting up Debezium CDC Publisher
[INFO] ════════════════════════════════════════════════════════════
[INFO] Executing Debezium config setup...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Debezium Config CDC Master Setup
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Environment:      web-universe-main-dev
[INFO]   PostgreSQL Host:  db-web-universe-main-dev-postgresql.fastorder.com
[INFO]   Database:         fastorder_web_universe_main_dev_db
[INFO]   Connector:        debezium-web-universe-main-dev-config
[INFO]   Topic Prefix:     cdc.web_universe_main_dev
[INFO]   Replication Slot: dbz_web_universe_main_dev_config
[INFO]   Tables:           config.public_defaults,config.feature_flags,config.config_version
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Step 1: Creating PostgreSQL publication for config tables...
[INFO] Creating publication: cdc_pub_web_config
CREATE PUBLICATION
[INFO] Setting REPLICA IDENTITY FULL for config tables...
ALTER TABLE
ALTER TABLE
ALTER TABLE
[INFO] Step 2: Creating replication slot...
[INFO] Creating replication slot: dbz_web_universe_main_dev_config
 (dbz_web_universe_main_dev_config,0/700E210)

[INFO] Step 3: Registering Debezium connector with Kafka Connect...
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps/02-setup-debezium-config.sh: line 232: [[: 0
0: syntax error in expression (error token is "0")
[INFO] Creating new connector: debezium-web-universe-main-dev-config
[INFO] Sending connector configuration to Kafka Connect...
[ERROR] Failed to register Debezium connector
[ERROR] Response: {"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nError while validating connector config: Could not load the private key\nYou can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`"}
[ OK ] Debezium config CDC master setup complete
[INFO] No topology.json found at /opt/fastorder/state/web-universe-main-dev/topology.json - skipping merge
[INFO] 
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Config CDC Setup Complete
[INFO] ═══════════════════════════════════════════════════════════════
═══════════════════════════════════════════════════════════════
  Service Capabilities: web-universe-main-dev
═══════════════════════════════════════════════════════════════

  Service:          web
  Zone:             universe
  Branch:           main
  Environment:      dev

  Config Schema:    ✅ YES
  Redis Cache:      ✅ YES
  CDC Role:         master

  CDC Master Configuration:
    Debezium:       debezium-web-universe-main-dev-config
    Topic Prefix:   cdc.web_universe_main_dev
    Repl Slot:      dbz_web_universe_main_dev_config
    Tables:         config.public_defaults,config.feature_flags,config.config_version

  Required Schemas: config tenant dashboard environment resource service item company communication ai
═══════════════════════════════════════════════════════════════
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] 
[INFO] Log file: /var/log/fastorder/cdc/config-cdc-20260205_054840.log
[ OK ] Config CDC setup finished successfully
[2026-02-05 05:49:04] ✅ Completed: 01-setup-config-cdc.sh
[2026-02-05 05:49:04] 
[2026-02-05 05:49:04] 🔧 Running: 02-setup-debezium-config.sh
[2026-02-05 05:49:04]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps/02-setup-debezium-config.sh
[2026-02-05 05:49:04]    Executing directly (script is executable)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Debezium Config CDC Master Setup
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   Environment:      web-universe-main-dev
[INFO]   PostgreSQL Host:  db-web-universe-main-dev-postgresql.fastorder.com
[INFO]   Database:         fastorder_web_universe_main_dev_db
[INFO]   Connector:        debezium-web-universe-main-dev-config
[INFO]   Topic Prefix:     cdc.web_universe_main_dev
[INFO]   Replication Slot: dbz_web_universe_main_dev_config
[INFO]   Tables:           config.public_defaults,config.feature_flags,config.config_version
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Step 1: Creating PostgreSQL publication for config tables...
[INFO] Publication cdc_pub_web_config already exists
[INFO] Setting REPLICA IDENTITY FULL for config tables...
ALTER TABLE
ALTER TABLE
ALTER TABLE
[INFO] Step 2: Creating replication slot...
[INFO] Replication slot dbz_web_universe_main_dev_config already exists
[INFO] Step 3: Registering Debezium connector with Kafka Connect...
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps/02-setup-debezium-config.sh: line 232: [[: 0
0: syntax error in expression (error token is "0")
[INFO] Creating new connector: debezium-web-universe-main-dev-config
[INFO] Sending connector configuration to Kafka Connect...
[ERROR] Failed to register Debezium connector
[ERROR] Response: {"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nError while validating connector config: Could not load the private key\nYou can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`"}
[2026-02-05 05:49:51] ❌ FAILED: 02-setup-debezium-config.sh (exit code: 1)
[2026-02-05 05:49:51] ❌ CRITICAL: This is a required step for CDC pipeline. Aborting.

[0;31m[ERROR][0m ❌ Database infrastructure (postgresql) setup failed with exit code: 1

📥 Download Full Logs

06-finalizing local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

Total Steps

Succeeded

Failed

Running

Pending

6 minutes

Total Steps Time

📊 Provisioning Job Status

⏱️ Timing Summary

📋 Job Details

📢 Viewing Old Job Attempt

🔄 Resume & Restart Options

📝 Execution Steps (9)