📊 Provisioning Job Status

[INFO] Using database engine from DB_ENGINE environment variable: postgresql
[INFO] Cleaning up any existing locks...

Starting database engine: postgresql
═══════════════════════════════════════════════

[INFO] Using environment from web interface: user-sau-main-dev
[2026-01-19 13:42:21] Using web-provided environment: user-sau-main-dev
[2026-01-19 13:42:21] Service: user, Zone: sau, Branch: main, Env: dev
✓ Environment initialized successfully (mode: general)
[INFO] Checking observability cell readiness: obs-user-sau-main-dev
[OK]   Observability cell endpoints registered for user-sau-main-dev
[INFO] Observability cell verified for user-sau-main-dev
[INFO] Monitoring will be configured after PostgreSQL deployment (step 10-monitoring-setup.sh)
[INFO] Citus mode ENABLED
[INFO] → Coordinator + 1 worker(s) + 1 standby node(s) per worker
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up coordinator (Citus control plane)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-19 13:42:22 UTC] USER=unknown EUID=33 PID=1363888 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-19 13:42:22 UTC] USER=unknown EUID=33 PID=1363895 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-19 13:42:22 UTC] USER=unknown EUID=33 PID=1363902 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-19 13:42:22 UTC] USER=unknown EUID=33 PID=1363909 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-19 13:42:22 UTC] USER=unknown EUID=33 PID=1363916 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-19 13:42:22 UTC] USER=unknown EUID=33 PID=1363923 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for user in sau-dev...
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566 JOB_UUID=fab5af05-2323-41ec-af35-3d55b2e9338d

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: coordinator
[INFO] PostgreSQL IP: 10.100.1.231
[INFO] Primary hostname: db-user-sau-main-dev-postgresql-coordinator.fastorder.com

[INFO] Adding /etc/hosts entries for coordinator...
[INFO]   1. db-user-sau-main-dev-postgresql.fastorder.com → 10.100.1.231 (primary/short)
[INFO]   2. db-user-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.231 (compatibility)

[INFO]   ✅ db-user-sau-main-dev-postgresql.fastorder.com already exists with correct IP
[INFO]   ✅ db-user-sau-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.231    db-user-sau-main-dev-postgresql-coordinator.fastorder.com
  10.100.1.231    db-user-sau-main-dev-postgresql.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-user-sau-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      user-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-user-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-19 13:42:25 UTC] USER=www-data EUID=0 PID=1364077 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator and /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:42:25 UTC] USER=www-data EUID=0 PID=1364086 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-19 13:42:26 UTC] USER=www-data EUID=0 PID=1364105 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1364039/ra_root.crt
[2026-01-19 13:42:26 UTC] USER=www-data EUID=0 PID=1364114 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1364039/ra_root.key
[2026-01-19 13:42:26 UTC] USER=www-data EUID=0 PID=1364123 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1364039/ra_root.crt
[2026-01-19 13:42:26 UTC] USER=www-data EUID=0 PID=1364145 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1364039/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-19 13:42:28 UTC] USER=www-data EUID=0 PID=1364218 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1364039/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-19 13:42:28 UTC] USER=www-data EUID=0 PID=1364227 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1364039/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-19 13:42:28 UTC] USER=www-data EUID=0 PID=1364236 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-19 13:42:28 UTC] USER=www-data EUID=0 PID=1364245 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1364039/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:28 UTC] USER=www-data EUID=0 PID=1364254 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:28 UTC] USER=www-data EUID=0 PID=1364263 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:28 UTC] USER=www-data EUID=0 PID=1364272 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-19 13:42:28 UTC] USER=www-data EUID=0 PID=1364283 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-19 13:42:28 UTC] USER=www-data EUID=0 PID=1364292 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-19 13:42:28 UTC] USER=www-data EUID=0 PID=1364301 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-19 13:42:28 UTC] USER=www-data EUID=0 PID=1364310 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-19 13:42:28 UTC] USER=www-data EUID=0 PID=1364319 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:42:28 UTC] USER=www-data EUID=0 PID=1364328 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: user-sau-main-dev
Node:        coordinator
Primary CN:  db-user-sau-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-user-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-user-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:42:29 UTC] USER=www-data EUID=0 PID=1364388 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-01-19 13:42:29 UTC] USER=www-data EUID=0 PID=1364397 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-19 13:42:29 UTC] USER=www-data EUID=0 PID=1364406 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-01-19 13:42:29 UTC] USER=www-data EUID=0 PID=1364416 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-19 13:42:29 UTC] USER=www-data EUID=0 PID=1364425 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364459 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364468 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364477 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364486 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364495 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364504 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364513 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364522 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364531 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364540 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364549 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364558 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364567 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364576 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364585 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364594 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364603 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:30 UTC] USER=www-data EUID=0 PID=1364612 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364638 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364647 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364657 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364675 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364685 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364702 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364723 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364734 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364743 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364752 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364761 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364770 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364780 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364790 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364799 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364808 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364817 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364826 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364835 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364844 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364853 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364863 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364873 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364882 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364895 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364905 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364915 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364924 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364934 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364943 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364952 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364961 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364970 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364979 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364988 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1364997 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1365006 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1365015 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1365025 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1365035 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1365044 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1365053 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1365062 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1365071 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1365080 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1365089 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-19 13:42:31 UTC] USER=www-data EUID=0 PID=1365098 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365107 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365116 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365125 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365134 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365145 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365155 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365164 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365173 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365182 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365191 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365200 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365209 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365218 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365227 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: postgres
Node: coordinator
FQDN: db-user-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-user-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365275 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365284 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365293 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365302 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365311 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365326 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:42:32 UTC] USER=www-data EUID=0 PID=1365336 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365356 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365370 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365388 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365407 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365422 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365433 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365443 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365452 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365471 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365481 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365490 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365500 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365509 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365527 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365548 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365557 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365566 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365594 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365603 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365612 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365621 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365630 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365639 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365658 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:33 UTC] USER=www-data EUID=0 PID=1365668 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365686 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365705 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365715 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365726 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365742 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365752 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365789 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365801 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365810 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365820 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365829 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365839 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365849 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365858 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365869 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365884 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365893 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365902 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365911 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365920 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365929 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365947 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365956 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:42:34 UTC] USER=www-data EUID=0 PID=1365966 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1365976 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1365985 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1365994 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1366003 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1366012 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1366021 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1366030 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1366048 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1366057 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1366066 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1366075 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1366085 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1366095 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1366105 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1366114 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1366123 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1366132 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1366141 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1366153 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:42:35 UTC] USER=www-data EUID=0 PID=1366174 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: postgres
Node: coordinator
FQDN: db-user-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-coordinator-postgresql environment: db-user-sau-main-dev-postgresql-coordinator.fastorder.com (10.100.1.231)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.231
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: coordinator
[INFO] Data dir:   /data/postgresql/17/user-sau-main-dev/coordinator
[INFO] Port:       5432
[INFO] Hostname:   db-user-sau-main-dev-postgresql-coordinator
[2026-01-19 13:42:37 UTC] USER=www-data EUID=0 PID=1366351 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:42:37 UTC] USER=www-data EUID=0 PID=1366372 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:42:37 UTC] USER=www-data EUID=0 PID=1366395 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:42:37 UTC] USER=www-data EUID=0 PID=1366424 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-user-sau-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      user-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-user-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-19 13:42:37 UTC] USER=www-data EUID=0 PID=1366465 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator and /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:42:37 UTC] USER=www-data EUID=0 PID=1366474 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-19 13:42:37 UTC] USER=www-data EUID=0 PID=1366484 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1366431
[2026-01-19 13:42:37 UTC] USER=www-data EUID=0 PID=1366493 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1366431/ra_root.crt
[2026-01-19 13:42:37 UTC] USER=www-data EUID=0 PID=1366502 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1366431/ra_root.key
[2026-01-19 13:42:37 UTC] USER=www-data EUID=0 PID=1366511 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1366431/ra_root.crt
[2026-01-19 13:42:37 UTC] USER=www-data EUID=0 PID=1366520 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1366431/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-19 13:42:39 UTC] USER=www-data EUID=0 PID=1366560 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1366431/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-19 13:42:39 UTC] USER=www-data EUID=0 PID=1366569 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1366431/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-19 13:42:39 UTC] USER=www-data EUID=0 PID=1366578 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-19 13:42:39 UTC] USER=www-data EUID=0 PID=1366587 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1366431/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:39 UTC] USER=www-data EUID=0 PID=1366596 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:39 UTC] USER=www-data EUID=0 PID=1366605 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:42:39 UTC] USER=www-data EUID=0 PID=1366614 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-19 13:42:39 UTC] USER=www-data EUID=0 PID=1366625 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-19 13:42:39 UTC] USER=www-data EUID=0 PID=1366634 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-19 13:42:39 UTC] USER=www-data EUID=0 PID=1366643 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-19 13:42:39 UTC] USER=www-data EUID=0 PID=1366652 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-19 13:42:39 UTC] USER=www-data EUID=0 PID=1366662 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:42:39 UTC] USER=www-data EUID=0 PID=1366671 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-user-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:user-sau-main-dev.fastorder.com, DNS:db-user-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:db-user-sau-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-user-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-user-sau-main-dev-postgresql.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: user-sau-main-dev
Node:        coordinator
Primary CN:  db-user-sau-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-user-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-19 13:42:39 UTC] USER=www-data EUID=0 PID=1366700 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-19 13:42:39 UTC] USER=www-data EUID=0 PID=1366709 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-19 13:42:39 UTC] USER=www-data EUID=0 PID=1366718 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-19 13:42:39 UTC] USER=www-data EUID=0 PID=1366739 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-coordinator.service
[2026-01-19 13:42:40 UTC] USER=www-data EUID=0 PID=1366767 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-user-sau-main-dev-coordinator
[OK]   No conflicting Postgres left on port 5432
[OK]   Using postgres password from vault provider
[2026-01-19 13:42:42 UTC] USER=www-data EUID=0 PID=1366901 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.X5DL8m
[2026-01-19 13:42:42 UTC] USER=www-data EUID=0 PID=1366924 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.X5DL8m
[2026-01-19 13:42:42 UTC] USER=www-data EUID=0 PID=1366946 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/user-sau-main-dev
[2026-01-19 13:42:42 UTC] USER=www-data EUID=0 PID=1366968 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev
[2026-01-19 13:42:42 UTC] USER=www-data EUID=0 PID=1366990 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/user-sau-main-dev
[INFO] Initializing cluster in /data/postgresql/17/user-sau-main-dev/coordinator (SCRAM; pwfile)
[WARN] Removing existing data directory: /data/postgresql/17/user-sau-main-dev/coordinator
[2026-01-19 13:42:42 UTC] USER=www-data EUID=0 PID=1367011 ACTION=fsop ARGS=rm -rf /data/postgresql/17/user-sau-main-dev/coordinator
[2026-01-19 13:42:42 UTC] USER=www-data EUID=0 PID=1367035 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/user-sau-main-dev/coordinator
[2026-01-19 13:42:42 UTC] USER=www-data EUID=0 PID=1367057 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev/coordinator
[2026-01-19 13:42:42 UTC] USER=www-data EUID=0 PID=1367103 ACTION=fsop ARGS=chmod 700 /data/postgresql/17/user-sau-main-dev/coordinator
[2026-01-19 13:42:42 UTC] USER=www-data EUID=0 PID=1367138 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-user-sau-main-dev-coordinator
[2026-01-19 13:42:42 UTC] USER=www-data EUID=0 PID=1367159 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-user-sau-main-dev-coordinator
[2026-01-19 13:42:42 UTC] USER=www-data EUID=0 PID=1367180 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-user-sau-main-dev-coordinator
[2026-01-19 13:42:42 UTC] USER=www-data EUID=0 PID=1367189 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/user-sau-main-dev/coordinator --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.X5DL8m
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /data/postgresql/17/user-sau-main-dev/coordinator ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/user-sau-main-dev/coordinator -l logfile start

[OK]   initdb complete
[2026-01-19 13:42:43 UTC] USER=www-data EUID=0 PID=1367250 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.X5DL8m
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-19 13:42:44 UTC] USER=www-data EUID=0 PID=1367300 ACTION=fsop ARGS=cp /tmp/tmp.2NeXEmBKMm /data/postgresql/17/user-sau-main-dev/coordinator/pg_hba.conf
[2026-01-19 13:42:44 UTC] USER=www-data EUID=0 PID=1367322 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev/coordinator/pg_hba.conf
[2026-01-19 13:42:44 UTC] USER=www-data EUID=0 PID=1367343 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/user-sau-main-dev/coordinator/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@user-sau-main-dev-coordinator.service
[2026-01-19 13:42:44 UTC] USER=www-data EUID=0 PID=1367368 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.YSy7AP /etc/systemd/system/postgresql@user-sau-main-dev-coordinator.service
[2026-01-19 13:42:44 UTC] USER=www-data EUID=0 PID=1367389 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@user-sau-main-dev-coordinator.service
[OK]   systemd unit written
[2026-01-19 13:42:44 UTC] USER=www-data EUID=0 PID=1367413 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-19 13:42:44 UTC] USER=www-data EUID=0 PID=1367434 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-19 13:42:44 UTC] USER=www-data EUID=0 PID=1367456 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-19 13:42:45 UTC] USER=www-data EUID=0 PID=1367640 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-coordinator.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-19 13:42:46 UTC] USER=www-data EUID=0 PID=1367696 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Provisioning application database and Debezium role (if not exists)...
[INFO] Checking if database fastorder_user_sau_main_dev_db exists...
[INFO] DB check result: exit_code=0, output='[2026-01-19 13:42:47 UTC] USER=www-data EUID=0 PID=1367924 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_user_sau_main_dev_db''
[INFO] Creating database fastorder_user_sau_main_dev_db...
[2026-01-19 13:42:47 UTC] USER=www-data EUID=0 PID=1367951 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_user_sau_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[OK]   Database fastorder_user_sau_main_dev_db created
[INFO] Checking if role debezium_user exists...
[INFO] Role check result: exit_code=0, output='[2026-01-19 13:42:48 UTC] USER=www-data EUID=0 PID=1367975 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[INFO] Creating role debezium_user...
[2026-01-19 13:42:48 UTC] USER=www-data EUID=0 PID=1368003 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'WeDe503By6lOFH4Cx6FWndxq';
CREATE ROLE
[OK]   Role debezium_user created
[2026-01-19 13:42:48 UTC] USER=www-data EUID=0 PID=1368027 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_user_sau_main_dev_db" TO debezium_user;
GRANT
[OK]   Application DB (fastorder_user_sau_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[INFO] Applying connection and memory optimizations...
[INFO] Current settings: max_connections=100, work_mem=4MB
[INFO] Target settings (coordinator): max_connections=150, work_mem=8MB
[2026-01-19 13:42:49 UTC] USER=www-data EUID=0 PID=1368108 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 150;
ALTER SYSTEM
[2026-01-19 13:42:49 UTC] USER=www-data EUID=0 PID=1368138 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-19 13:42:49 UTC] USER=www-data EUID=0 PID=1368163 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   Settings applied to postgresql.auto.conf
[2026-01-19 13:42:49 UTC] USER=www-data EUID=0 PID=1368178 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/user-sau-main-dev/coordinator/standby.signal
[INFO] Service recently started (3s ago) - restarting to apply max_connections...
[INFO] Stopping service...
[2026-01-19 13:42:49 UTC] USER=www-data EUID=0 PID=1368209 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-coordinator.service
[INFO] Waiting for port 5432 to be released...
[OK]   Port 5432 released
[INFO] Starting service...
[2026-01-19 13:42:53 UTC] USER=www-data EUID=0 PID=1368324 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-coordinator.service
[2026-01-19 13:42:58 UTC] USER=www-data EUID=0 PID=1368550 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[OK]   ✅ Optimization complete: max_connections=150, work_mem=8MB
[INFO] Setting postgres password via centralized script... for coordinator
[INFO] Temporarily disabling synchronous_commit on coordinator for password setting...
[OK]   Disabled synchronous_commit (was: on)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    user
  Zone:       sau
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/user/sau/main/dev/postgresql/coordinator

Connection Info:
  Socket Dir: /var/run/postgresql-user-sau-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Initial setup: Using password from initdb
✓ PostgreSQL password already set during initdb
Storing password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/coordinator
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/user/sau/main/dev/postgresql/coordinator
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
[INFO] Restoring synchronous_commit on coordinator...
[OK]   Restored synchronous_commit to: on
[OK]   Password set and persisted
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: coordinator
[INFO] PostgreSQL IP: 10.100.1.231
[INFO] Primary hostname: db-user-sau-main-dev-postgresql-coordinator.fastorder.com

[INFO] Adding /etc/hosts entries for coordinator...
[INFO]   1. db-user-sau-main-dev-postgresql.fastorder.com → 10.100.1.231 (primary/short)
[INFO]   2. db-user-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.231 (compatibility)

[INFO]   ✅ db-user-sau-main-dev-postgresql.fastorder.com already exists with correct IP
[INFO]   ✅ db-user-sau-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.231    db-user-sau-main-dev-postgresql-coordinator.fastorder.com
  10.100.1.231    db-user-sau-main-dev-postgresql.fastorder.com


[OK]   PostgreSQL 'user-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key \
        host=db-user-sau-main-dev-postgresql-coordinator port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        user-sau-main-dev-postgresql-coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.231
[INFO]   Port:              5432
[INFO]   FQDN:              db-user-sau-main-dev-postgresql-coordinator
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 83d5bc7d-3699-4f7e-98b2-72fdfea60e05
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 📦 03 role...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-19 13:43:10 UTC] USER=www-data EUID=0 PID=1369255 ACTION=fsop ARGS=test -f /data/postgresql/17/user-sau-main-dev/coordinator/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    debezium_user
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   debezium_user
  Hostname:    db-user-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:43:11 UTC] USER=www-data EUID=0 PID=1369430 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-debezium_user
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369439 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369448 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-debezium_user/ra_root.key
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369457 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369466 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369500 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369509 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369518 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369527 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369536 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369545 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369554 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369563 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369572 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369581 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369591 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369600 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369609 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369618 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369627 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369636 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369645 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369671 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369680 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369698 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:12 UTC] USER=www-data EUID=0 PID=1369716 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369734 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369743 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369752 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369761 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369770 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_pk8.der
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369780 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369790 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369799 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369817 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369828 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369837 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369846 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369855 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369864 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369873 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369883 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369892 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_pk8.der
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369903 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369913 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369922 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369941 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369952 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369969 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369979 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369988 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1369997 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1370006 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-19 13:43:13 UTC] USER=www-data EUID=0 PID=1370015 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370024 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_pk8.der
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370039 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370055 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370064 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370073 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370082 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370092 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370101 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370110 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370119 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370128 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370137 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370146 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370155 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_pk8.der
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370165 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370175 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370184 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370193 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370202 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370211 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370220 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370229 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370238 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:43:14 UTC] USER=www-data EUID=0 PID=1370247 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: debezium_user
Node: coordinator
FQDN: db-user-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-coordinator.fastorder.com -U debezium_user -d postgres

✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    user
  Zone:       sau
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd

Connection Info:
  Socket Dir: /var/run/postgresql-user-sau-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: fastorder_admin_gd
Storing password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: coordinator/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-user-sau-main-dev-postgresql.fastorder.com
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    fastorder_admin_gd
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   fastorder_admin_gd
  Hostname:    db-user-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:43:22 UTC] USER=www-data EUID=0 PID=1370765 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-fastorder_admin_gd
[2026-01-19 13:43:22 UTC] USER=www-data EUID=0 PID=1370774 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-19 13:43:22 UTC] USER=www-data EUID=0 PID=1370784 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
[2026-01-19 13:43:22 UTC] USER=www-data EUID=0 PID=1370793 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-19 13:43:22 UTC] USER=www-data EUID=0 PID=1370802 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370821 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370830 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370839 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370848 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370857 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370866 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370875 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370884 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370893 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370902 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370911 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370921 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370930 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370940 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370949 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370958 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370967 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370976 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370985 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1370995 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1371004 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1371030 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1371039 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1371048 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1371057 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1371066 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1371075 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1371084 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1371093 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1371102 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1371111 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:43:23 UTC] USER=www-data EUID=0 PID=1371120 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371129 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371139 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371158 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371167 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371178 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371187 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371196 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371205 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371214 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371223 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371232 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371241 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371250 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371259 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371269 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371280 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371289 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371298 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371307 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371329 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371345 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371355 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371364 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371373 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371382 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371391 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371401 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371411 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371420 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371429 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371438 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371447 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371457 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371466 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371475 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371484 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371493 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:43:24 UTC] USER=www-data EUID=0 PID=1371502 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-19 13:43:25 UTC] USER=www-data EUID=0 PID=1371511 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-19 13:43:25 UTC] USER=www-data EUID=0 PID=1371521 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:43:25 UTC] USER=www-data EUID=0 PID=1371531 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:43:25 UTC] USER=www-data EUID=0 PID=1371540 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:43:25 UTC] USER=www-data EUID=0 PID=1371549 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:43:25 UTC] USER=www-data EUID=0 PID=1371558 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:43:25 UTC] USER=www-data EUID=0 PID=1371567 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:43:25 UTC] USER=www-data EUID=0 PID=1371576 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:43:25 UTC] USER=www-data EUID=0 PID=1371585 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:43:25 UTC] USER=www-data EUID=0 PID=1371594 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:43:25 UTC] USER=www-data EUID=0 PID=1371603 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: fastorder_admin_gd
Node: coordinator
FQDN: db-user-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-coordinator.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-user-sau-main-dev-coordinator:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_user_sau_main_dev_db already exists, skipping creation
[2026-01-19 13:43:25 UTC] USER=www-data EUID=0 PID=1371661 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /data/postgresql/17/user-sau-main-dev/coordinator/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-19 13:43:25 UTC] USER=www-data EUID=0 PID=1371701 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-coordinator.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

=== Pre-flight Checks ===
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
✓ AWS Secrets Manager accessible

=== Retrieving Credentials from AWS ===
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials loaded for coordinator/fastorder_admin_gd: fastorder_admin_gd@db-user-sau-main-dev-postgresql.fastorder.com:5432/fastorder_user_sau_main_dev_db
✓ Credentials retrieved: fastorder_admin_gd@db-user-sau-main-dev-postgresql.fastorder.com:5432/fastorder_user_sau_main_dev_db
╔════════════════════════════════════════════╗
║  PostgreSQL Test Suite (AWS Secrets MGR)  ║
╚════════════════════════════════════════════╝

=== PostgreSQL Authentication Test ===
✗ PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-user-sau-main-dev-postgresql.fastorder.com" (10.100.1.231), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-19 13:43:30 UTC] USER=www-data EUID=0 PID=1372040 ACTION=fsop ARGS=test -f /data/postgresql/17/user-sau-main-dev/coordinator/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : user-sau-main-dev
  IDENTIFIER  : coordinator
  PG HOST     : db-user-sau-main-dev-postgresql.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_user_sau_main_dev_db
  SCHEMA      : user
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
  DNS → 10.100.1.231
  CA         : /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    user
  Zone:       sau
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user

Connection Info:
  Socket Dir: /var/run/postgresql-user-sau-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User debezium_user does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: debezium_user
Storing password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: coordinator/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-01-19 13:43:38 UTC] USER=www-data EUID=0 PID=1372530 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_user_sau_main_dev_db already exists
[2026-01-19 13:43:38 UTC] USER=www-data EUID=0 PID=1372557 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d fastorder_user_sau_main_dev_db --no-psqlrc
ERROR:  syntax error at or near "user"
LINE 1: CREATE SCHEMA IF NOT EXISTS user;
                                    ^
GRANT
ERROR:  syntax error at or near "user"
LINE 1: GRANT USAGE ON SCHEMA user TO debezium_user;
                              ^
ERROR:  syntax error at or near "user"
LINE 1: GRANT SELECT ON ALL TABLES IN SCHEMA user TO debezium_user;
                                             ^
ERROR:  syntax error at or near "user"
LINE 1: GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA user TO debez...
                                                       ^
ERROR:  syntax error at or near "user"
LINE 1: ALTER DEFAULT PRIVILEGES IN SCHEMA user GRANT SELECT ON TABL...
                                           ^
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/user-sau-main-dev/coordinator/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (user) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=user
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/user/run.sh
🔍 DEBUG_CHECKPOINT_03: No specific folder for user, using default
[DEBUG] Tracking substep start: steps/01-install/steps/default (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 🔸 Service: user (using default contracts schema)
🔍 DEBUG_CHECKPOINT_04: Executing default: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/run.sh
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting default contracts schema provisioning for SERVICE=user
[INFO] Environment: user-sau-main-dev
[INFO] Schema: user (contracts tables)
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG: Looking for contracts steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/default/contracts/steps
[INFO] 📁 Running contracts schema setup for: user
[INFO] 📁 Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/default/contracts/steps

[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing user schema (contracts tables)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Schema:      user
  Identifier:  coordinator
  Database:    fastorder_user_sau_main_dev_db
  Host:        db-user-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_user_sau_main_dev_db
ℹ️  Database fastorder_user_sau_main_dev_db already exists
✅ Connected to database: fastorder_user_sau_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
CREATE EXTENSION
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating user schema...
CREATE SCHEMA
✅ Schema created
🔧 Creating contracts tables in user schema...
   Creating "user".contract_key...
CREATE TABLE
   Creating "user".contract_type...
CREATE TABLE
   Creating "user".contracts...
CREATE TABLE
   Adding columns to "user".contracts (safe migration)...
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
UPDATE 0
UPDATE 0
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
   Creating "user".contract_vars...
CREATE TABLE
   Creating "user".contract_datetime...
CREATE TABLE
   Creating "user".contract_decimal...
CREATE TABLE
   Creating "user".contract_float...
CREATE TABLE
   Creating "user".contract_int...
CREATE TABLE
   Creating "user".contract_json...
CREATE TABLE
   Creating "user".contract_terms...
CREATE TABLE
   Creating "user".contract_term_contracts...
CREATE TABLE
   Creating "user".contract_term_datetime...
CREATE TABLE
   Creating "user".contract_term_decimal...
CREATE TABLE
   Creating "user".contract_term_float...
CREATE TABLE
   Creating "user".contract_term_int...
CREATE TABLE
   Creating "user".contract_term_items...
CREATE TABLE
   Creating "user".contract_term_json...
CREATE TABLE
   Creating "user".contract_term_vars...
CREATE TABLE
   Creating "user".user_id_uuid_mapping...
CREATE TABLE
✅ All 19 tables created
🔧 Creating indexes...
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
✅ All indexes created
🔧 Creating foreign keys...
DO
DO
✅ Foreign keys created
🔧 Configuring Citus distribution...
   Creating reference table: contract_key
 create_reference_table 
------------------------
 
(1 row)

   Creating reference table: contract_type
 create_reference_table 
------------------------
 
(1 row)

   Creating distributed table: contracts
   Creating distributed table: contract_vars
   Creating distributed table: contract_datetime
   Creating distributed table: contract_decimal
   Creating distributed table: contract_float
   Creating distributed table: contract_int
   Creating distributed table: contract_json
   Creating distributed table: contract_terms
   Creating distributed table: contract_term_contracts
   Creating distributed table: contract_term_datetime
   Creating distributed table: contract_term_decimal
   Creating distributed table: contract_term_float
   Creating distributed table: contract_term_int
   Creating distributed table: contract_term_items
   Creating distributed table: contract_term_json
 create_distributed_table 
--------------------------
 
(1 row)

   Creating distributed table: contract_term_vars
 create_distributed_table 
--------------------------
 
(1 row)

✅ Citus distribution configured
🎉 Schema initialization complete for user in fastorder_user_sau_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

==========================================
✅ user schema initialization complete!
   Tables: 19
   Indexes: 54
==========================================

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Default contracts schema setup complete for: user
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

✓ ✅ Coordinator setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up 1 worker(s) (Citus data nodes)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up worker: worker-01
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-19 13:43:56 UTC] USER=unknown EUID=33 PID=1373398 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-19 13:43:56 UTC] USER=unknown EUID=33 PID=1373405 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-19 13:43:56 UTC] USER=unknown EUID=33 PID=1373412 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-19 13:43:56 UTC] USER=unknown EUID=33 PID=1373419 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-19 13:43:56 UTC] USER=unknown EUID=33 PID=1373426 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-19 13:43:56 UTC] USER=unknown EUID=33 PID=1373434 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for user in sau-dev...
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566 JOB_UUID=fab5af05-2323-41ec-af35-3d55b2e9338d

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.232
[INFO] Primary hostname: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01...
[INFO]   db-user-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.232

[INFO]   ✅ db-user-sau-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.232    db-user-sau-main-dev-postgresql-worker-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-user-sau-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      user-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-19 13:43:59 UTC] USER=www-data EUID=0 PID=1373692 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01 and /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:43:59 UTC] USER=www-data EUID=0 PID=1373701 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-19 13:43:59 UTC] USER=www-data EUID=0 PID=1373711 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1373657
[2026-01-19 13:43:59 UTC] USER=www-data EUID=0 PID=1373721 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1373657/ra_root.crt
[2026-01-19 13:43:59 UTC] USER=www-data EUID=0 PID=1373730 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1373657/ra_root.key
[2026-01-19 13:43:59 UTC] USER=www-data EUID=0 PID=1373739 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1373657/ra_root.crt
[2026-01-19 13:43:59 UTC] USER=www-data EUID=0 PID=1373749 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1373657/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[2026-01-19 13:44:01 UTC] USER=www-data EUID=0 PID=1373821 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1373657/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-19 13:44:01 UTC] USER=www-data EUID=0 PID=1373845 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1373657/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-19 13:44:01 UTC] USER=www-data EUID=0 PID=1373858 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-19 13:44:01 UTC] USER=www-data EUID=0 PID=1373867 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1373657/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:01 UTC] USER=www-data EUID=0 PID=1373876 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:01 UTC] USER=www-data EUID=0 PID=1373885 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:02 UTC] USER=www-data EUID=0 PID=1373894 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-19 13:44:02 UTC] USER=www-data EUID=0 PID=1373907 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-19 13:44:02 UTC] USER=www-data EUID=0 PID=1373917 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-19 13:44:02 UTC] USER=www-data EUID=0 PID=1373926 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-19 13:44:02 UTC] USER=www-data EUID=0 PID=1373935 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-19 13:44:02 UTC] USER=www-data EUID=0 PID=1373944 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-user-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:user-sau-main-dev.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: user-sau-main-dev
Node:        worker-01
Primary CN:  db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-user-sau-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:44:02 UTC] USER=www-data EUID=0 PID=1374052 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-01-19 13:44:02 UTC] USER=www-data EUID=0 PID=1374061 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-19 13:44:02 UTC] USER=www-data EUID=0 PID=1374070 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-01-19 13:44:02 UTC] USER=www-data EUID=0 PID=1374079 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-19 13:44:02 UTC] USER=www-data EUID=0 PID=1374094 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374136 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374145 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374154 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374172 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374181 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374190 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374201 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374210 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374227 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374236 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374245 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374254 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374263 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374272 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374281 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374290 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374316 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374325 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374334 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374343 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374352 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374361 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374375 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374397 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374406 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374415 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:44:03 UTC] USER=www-data EUID=0 PID=1374424 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374434 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374444 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374454 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374464 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374473 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374482 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374491 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374500 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374509 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374518 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374527 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374536 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374545 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374555 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374565 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374576 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374585 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374596 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374605 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374614 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374623 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374635 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374645 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374660 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374673 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374682 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374692 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374702 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:04 UTC] USER=www-data EUID=0 PID=1374711 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374720 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374730 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374740 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374749 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374758 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374767 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374776 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374785 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374795 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374804 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374814 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374828 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374838 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374854 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374863 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374873 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374882 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374891 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374900 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374909 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: postgres
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374952 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-01-19 13:44:05 UTC] USER=www-data EUID=0 PID=1374963 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1374972 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1374981 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1374991 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375010 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375021 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375034 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375047 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375056 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375065 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375078 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375092 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375102 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375111 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375120 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375129 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375138 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375147 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375156 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375165 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375174 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375183 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375192 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375201 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:06 UTC] USER=www-data EUID=0 PID=1375210 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375243 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375252 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375262 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375271 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375280 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375289 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375298 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375311 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375320 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375329 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375338 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375347 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375357 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375367 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375376 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375385 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375394 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375403 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375412 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375421 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375430 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375439 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375448 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375457 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375466 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375476 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375486 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375495 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375504 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375513 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375522 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375531 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375540 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375549 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375558 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375567 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375576 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375585 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375595 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375605 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:07 UTC] USER=www-data EUID=0 PID=1375614 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375626 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375635 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375644 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375653 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375672 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375693 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375702 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375711 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375720 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375730 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375740 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375749 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375758 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375767 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375776 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375785 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375795 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375807 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:44:08 UTC] USER=www-data EUID=0 PID=1375817 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: postgres
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-postgresql environment: db-user-sau-main-dev-postgresql-worker-01.fastorder.com (10.100.1.232)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.232
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: worker-01
[INFO] Data dir:   /data/postgresql/17/user-sau-main-dev/worker-01
[INFO] Port:       5432
[INFO] Hostname:   db-user-sau-main-dev-postgresql-worker-01
[2026-01-19 13:44:10 UTC] USER=www-data EUID=0 PID=1376017 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:10 UTC] USER=www-data EUID=0 PID=1376038 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:10 UTC] USER=www-data EUID=0 PID=1376059 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-user-sau-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      user-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-19 13:44:10 UTC] USER=www-data EUID=0 PID=1376149 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01 and /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:10 UTC] USER=www-data EUID=0 PID=1376158 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-19 13:44:10 UTC] USER=www-data EUID=0 PID=1376168 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1376066
[2026-01-19 13:44:10 UTC] USER=www-data EUID=0 PID=1376177 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1376066/ra_root.crt
[2026-01-19 13:44:10 UTC] USER=www-data EUID=0 PID=1376189 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1376066/ra_root.key
[2026-01-19 13:44:10 UTC] USER=www-data EUID=0 PID=1376199 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1376066/ra_root.crt
[2026-01-19 13:44:10 UTC] USER=www-data EUID=0 PID=1376210 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1376066/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[2026-01-19 13:44:12 UTC] USER=www-data EUID=0 PID=1376271 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1376066/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-19 13:44:12 UTC] USER=www-data EUID=0 PID=1376280 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1376066/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-19 13:44:12 UTC] USER=www-data EUID=0 PID=1376289 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-19 13:44:12 UTC] USER=www-data EUID=0 PID=1376298 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1376066/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:12 UTC] USER=www-data EUID=0 PID=1376307 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:12 UTC] USER=www-data EUID=0 PID=1376316 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-19 13:44:12 UTC] USER=www-data EUID=0 PID=1376339 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-19 13:44:12 UTC] USER=www-data EUID=0 PID=1376351 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-19 13:44:12 UTC] USER=www-data EUID=0 PID=1376360 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-19 13:44:12 UTC] USER=www-data EUID=0 PID=1376369 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-19 13:44:12 UTC] USER=www-data EUID=0 PID=1376378 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:12 UTC] USER=www-data EUID=0 PID=1376387 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-user-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:user-sau-main-dev.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: user-sau-main-dev
Node:        worker-01
Primary CN:  db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-user-sau-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-19 13:44:12 UTC] USER=www-data EUID=0 PID=1376416 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-19 13:44:12 UTC] USER=www-data EUID=0 PID=1376427 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-19 13:44:12 UTC] USER=www-data EUID=0 PID=1376440 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-19 13:44:13 UTC] USER=www-data EUID=0 PID=1376472 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-worker-01.service
[2026-01-19 13:44:13 UTC] USER=www-data EUID=0 PID=1376510 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-user-sau-main-dev-worker-01
[OK]   No conflicting Postgres left on port 5432
[OK]   Using postgres password from vault provider
[2026-01-19 13:44:15 UTC] USER=www-data EUID=0 PID=1376636 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.bZZbAf
[2026-01-19 13:44:15 UTC] USER=www-data EUID=0 PID=1376658 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/user-sau-main-dev
[2026-01-19 13:44:15 UTC] USER=www-data EUID=0 PID=1376704 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/user-sau-main-dev
[INFO] Initializing cluster in /data/postgresql/17/user-sau-main-dev/worker-01 (SCRAM; pwfile)
[WARN] Removing existing data directory: /data/postgresql/17/user-sau-main-dev/worker-01
[2026-01-19 13:44:15 UTC] USER=www-data EUID=0 PID=1376727 ACTION=fsop ARGS=rm -rf /data/postgresql/17/user-sau-main-dev/worker-01
[2026-01-19 13:44:15 UTC] USER=www-data EUID=0 PID=1376770 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/user-sau-main-dev/worker-01
[2026-01-19 13:44:16 UTC] USER=www-data EUID=0 PID=1376817 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev/worker-01
[2026-01-19 13:44:16 UTC] USER=www-data EUID=0 PID=1376844 ACTION=fsop ARGS=chmod 700 /data/postgresql/17/user-sau-main-dev/worker-01
[2026-01-19 13:44:16 UTC] USER=www-data EUID=0 PID=1376866 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-user-sau-main-dev-worker-01
[2026-01-19 13:44:16 UTC] USER=www-data EUID=0 PID=1376909 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-user-sau-main-dev-worker-01
[2026-01-19 13:44:16 UTC] USER=www-data EUID=0 PID=1376919 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/user-sau-main-dev/worker-01 --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.bZZbAf
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /data/postgresql/17/user-sau-main-dev/worker-01 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/user-sau-main-dev/worker-01 -l logfile start

[OK]   initdb complete
[2026-01-19 13:44:17 UTC] USER=www-data EUID=0 PID=1376968 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.bZZbAf
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-19 13:44:18 UTC] USER=www-data EUID=0 PID=1377017 ACTION=fsop ARGS=cp /tmp/tmp.8Kd8TBh8GW /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[2026-01-19 13:44:18 UTC] USER=www-data EUID=0 PID=1377048 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[2026-01-19 13:44:18 UTC] USER=www-data EUID=0 PID=1377070 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@user-sau-main-dev-worker-01.service
[2026-01-19 13:44:18 UTC] USER=www-data EUID=0 PID=1377125 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@user-sau-main-dev-worker-01.service
[OK]   systemd unit written
[2026-01-19 13:44:18 UTC] USER=www-data EUID=0 PID=1377167 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-19 13:44:18 UTC] USER=www-data EUID=0 PID=1377188 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-19 13:44:20 UTC] USER=www-data EUID=0 PID=1377329 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-worker-01.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-19 13:44:20 UTC] USER=www-data EUID=0 PID=1377373 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-worker-01.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Provisioning application database and Debezium role (if not exists)...
[INFO] Checking if database fastorder_user_sau_main_dev_db exists...
[INFO] DB check result: exit_code=0, output='[2026-01-19 13:44:21 UTC] USER=www-data EUID=0 PID=1377552 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_user_sau_main_dev_db''
[INFO] Creating database fastorder_user_sau_main_dev_db...
[2026-01-19 13:44:21 UTC] USER=www-data EUID=0 PID=1377576 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_user_sau_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[OK]   Database fastorder_user_sau_main_dev_db created
[INFO] Checking if role debezium_user exists...
[INFO] Role check result: exit_code=0, output='[2026-01-19 13:44:22 UTC] USER=www-data EUID=0 PID=1377601 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[INFO] Creating role debezium_user...
[2026-01-19 13:44:22 UTC] USER=www-data EUID=0 PID=1377629 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'zttOjtRKfiGBUr7pKtEOSBEC';
CREATE ROLE
[OK]   Role debezium_user created
[2026-01-19 13:44:22 UTC] USER=www-data EUID=0 PID=1377654 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_user_sau_main_dev_db" TO debezium_user;
GRANT
[OK]   Application DB (fastorder_user_sau_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[INFO] Applying connection and memory optimizations...
[INFO] Current settings: max_connections=100, work_mem=4MB
[INFO] Target settings (worker): max_connections=100, work_mem=8MB
[2026-01-19 13:44:23 UTC] USER=www-data EUID=0 PID=1377748 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 100;
ALTER SYSTEM
[2026-01-19 13:44:23 UTC] USER=www-data EUID=0 PID=1377771 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-19 13:44:23 UTC] USER=www-data EUID=0 PID=1377808 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   Settings applied to postgresql.auto.conf
[INFO] Service recently started (3s ago) - restarting to apply max_connections...
[INFO] Stopping service...
[2026-01-19 13:44:23 UTC] USER=www-data EUID=0 PID=1377847 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-worker-01.service
[INFO] Waiting for port 5432 to be released...
[OK]   Port 5432 released
[INFO] Starting service...
[2026-01-19 13:44:26 UTC] USER=www-data EUID=0 PID=1377927 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-worker-01.service
[2026-01-19 13:44:32 UTC] USER=www-data EUID=0 PID=1378091 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-worker-01.service
[OK]   ✅ Optimization complete: max_connections=100, work_mem=8MB
[OK]   Synchronous replication already configured (synchronous_commit: on)
[INFO] Setting postgres password via centralized script... for worker-01
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    user
  Zone:       sau
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/user/sau/main/dev/postgresql/worker-01

Connection Info:
  Socket Dir: /var/run/postgresql-user-sau-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Initial setup: Using password from initdb
✓ PostgreSQL password already set during initdb
Storing password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/worker-01
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/user/sau/main/dev/postgresql/worker-01
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
[OK]   Password set and persisted
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.232
[INFO] Primary hostname: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01...
[INFO]   db-user-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.232

[INFO]   ✅ db-user-sau-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.232    db-user-sau-main-dev-postgresql-worker-01.fastorder.com


[OK]   PostgreSQL 'user-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key \
        host=db-user-sau-main-dev-postgresql-worker-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        user-sau-main-dev-postgresql-worker-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.232
[INFO]   Port:              5432
[INFO]   FQDN:              db-user-sau-main-dev-postgresql-worker-01
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 87ccba48-d8e0-43e4-97b8-d87917a5d35c
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 📦 03 role...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-19 13:44:42 UTC] USER=www-data EUID=0 PID=1378624 ACTION=fsop ARGS=test -f /data/postgresql/17/user-sau-main-dev/worker-01/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    debezium_user
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   debezium_user
  Hostname:    db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:44:43 UTC] USER=www-data EUID=0 PID=1378779 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-debezium_user
[2026-01-19 13:44:43 UTC] USER=www-data EUID=0 PID=1378788 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-01-19 13:44:43 UTC] USER=www-data EUID=0 PID=1378797 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-debezium_user/ra_root.key
[2026-01-19 13:44:43 UTC] USER=www-data EUID=0 PID=1378806 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-01-19 13:44:43 UTC] USER=www-data EUID=0 PID=1378815 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:43 UTC] USER=www-data EUID=0 PID=1378832 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:43 UTC] USER=www-data EUID=0 PID=1378841 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:43 UTC] USER=www-data EUID=0 PID=1378850 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-19 13:44:43 UTC] USER=www-data EUID=0 PID=1378859 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt
[2026-01-19 13:44:43 UTC] USER=www-data EUID=0 PID=1378868 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1378877 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1378894 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1378909 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1378918 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_pk8.der
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1378927 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1378936 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1378945 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1378954 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1378963 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1378973 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1378982 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_pk8.der
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1378991 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379000 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379026 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379035 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379044 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379053 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379062 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379071 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379080 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379089 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379098 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379107 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379116 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379125 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_pk8.der
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379135 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379146 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379155 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379164 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379173 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379182 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379193 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379202 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379219 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379228 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379237 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379246 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379255 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_pk8.der
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379265 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:44 UTC] USER=www-data EUID=0 PID=1379275 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379284 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379293 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379302 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379311 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379320 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379329 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379338 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379347 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379356 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379365 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379374 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_pk8.der
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379384 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379395 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379404 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379413 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379423 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379434 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379452 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379461 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379470 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379479 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379488 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379497 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379506 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_pk8.der
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379517 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379528 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379537 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379546 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379556 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379565 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379574 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379584 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379594 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:44:45 UTC] USER=www-data EUID=0 PID=1379603 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: debezium_user
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U debezium_user -d postgres

🔐 Generating replicator client certificate for worker-01...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379644 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379655 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379664 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379673 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379683 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379698 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379707 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379716 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379725 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379734 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379743 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379753 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379763 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379774 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379785 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379802 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379816 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379836 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379851 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379869 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379879 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379889 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379898 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379908 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379917 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:46 UTC] USER=www-data EUID=0 PID=1379926 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1379952 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1379961 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1379970 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1379979 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1379988 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1379997 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380006 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380015 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380024 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380033 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380042 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380051 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380062 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380072 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380082 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380092 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380101 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380119 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380139 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380148 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380157 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380166 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380175 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380185 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380195 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380204 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380213 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380222 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380231 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380242 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380254 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380268 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380277 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380286 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:44:47 UTC] USER=www-data EUID=0 PID=1380295 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380314 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380324 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380333 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380342 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380351 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380367 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380377 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380397 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380406 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380415 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380424 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380443 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380454 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380463 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
   ✅ Symlinked ca.pem
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380481 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380490 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380500 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380510 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380519 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:44:48 UTC] USER=www-data EUID=0 PID=1380528 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

✅ Replicator certificate generated for worker-01
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    user
  Zone:       sau
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd

Connection Info:
  Socket Dir: /var/run/postgresql-user-sau-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: fastorder_admin_gd
Storing password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: worker-01/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    fastorder_admin_gd
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   fastorder_admin_gd
  Hostname:    db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:44:56 UTC] USER=www-data EUID=0 PID=1381110 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-fastorder_admin_gd
[2026-01-19 13:44:56 UTC] USER=www-data EUID=0 PID=1381119 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-01-19 13:44:56 UTC] USER=www-data EUID=0 PID=1381137 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-01-19 13:44:56 UTC] USER=www-data EUID=0 PID=1381146 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381175 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381184 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381193 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381202 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381211 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381220 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381229 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381238 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381247 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381256 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381266 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381276 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381294 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381303 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381312 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381321 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381330 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-19 13:44:57 UTC] USER=www-data EUID=0 PID=1381348 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:58 UTC] USER=www-data EUID=0 PID=1381357 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:58 UTC] USER=www-data EUID=0 PID=1381384 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:58 UTC] USER=www-data EUID=0 PID=1381394 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:58 UTC] USER=www-data EUID=0 PID=1381413 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:58 UTC] USER=www-data EUID=0 PID=1381431 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-19 13:44:58 UTC] USER=www-data EUID=0 PID=1381449 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:58 UTC] USER=www-data EUID=0 PID=1381458 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:58 UTC] USER=www-data EUID=0 PID=1381467 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:44:58 UTC] USER=www-data EUID=0 PID=1381489 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-01-19 13:44:58 UTC] USER=www-data EUID=0 PID=1381528 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:58 UTC] USER=www-data EUID=0 PID=1381555 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:58 UTC] USER=www-data EUID=0 PID=1381592 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:58 UTC] USER=www-data EUID=0 PID=1381622 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:58 UTC] USER=www-data EUID=0 PID=1381631 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:58 UTC] USER=www-data EUID=0 PID=1381650 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381659 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381668 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381686 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381695 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381707 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381717 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381727 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381736 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381745 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381754 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381772 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381781 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381810 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381819 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381830 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381839 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381849 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381859 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381868 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381877 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381895 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381904 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381913 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-19 13:44:59 UTC] USER=www-data EUID=0 PID=1381927 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:45:00 UTC] USER=www-data EUID=0 PID=1381945 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:45:00 UTC] USER=www-data EUID=0 PID=1381965 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:45:00 UTC] USER=www-data EUID=0 PID=1381981 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-19 13:45:00 UTC] USER=www-data EUID=0 PID=1381999 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-01-19 13:45:00 UTC] USER=www-data EUID=0 PID=1382010 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:45:00 UTC] USER=www-data EUID=0 PID=1382020 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:45:00 UTC] USER=www-data EUID=0 PID=1382032 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:45:00 UTC] USER=www-data EUID=0 PID=1382057 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:45:00 UTC] USER=www-data EUID=0 PID=1382075 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:45:00 UTC] USER=www-data EUID=0 PID=1382084 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:45:00 UTC] USER=www-data EUID=0 PID=1382095 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:45:00 UTC] USER=www-data EUID=0 PID=1382104 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: fastorder_admin_gd
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-user-sau-main-dev-worker-01:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_user_sau_main_dev_db already exists, skipping creation
[2026-01-19 13:45:01 UTC] USER=www-data EUID=0 PID=1382173 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-19 13:45:01 UTC] USER=www-data EUID=0 PID=1382214 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-worker-01.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

=== Pre-flight Checks ===
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
✓ AWS Secrets Manager accessible

=== Retrieving Credentials from AWS ===
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials loaded for worker-01/fastorder_admin_gd: fastorder_admin_gd@db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_user_sau_main_dev_db
✓ Credentials retrieved: fastorder_admin_gd@db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_user_sau_main_dev_db
╔════════════════════════════════════════════╗
║  PostgreSQL Test Suite (AWS Secrets MGR)  ║
╚════════════════════════════════════════════╝

=== PostgreSQL Authentication Test ===
✗ PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-user-sau-main-dev-postgresql-worker-01.fastorder.com" (10.100.1.232), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-19 13:45:09 UTC] USER=www-data EUID=0 PID=1384751 ACTION=fsop ARGS=test -f /data/postgresql/17/user-sau-main-dev/worker-01/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : user-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_user_sau_main_dev_db
  SCHEMA      : user
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
  DNS → 10.100.1.232
  CA         : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    user
  Zone:       sau
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user

Connection Info:
  Socket Dir: /var/run/postgresql-user-sau-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User debezium_user does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: debezium_user
Storing password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: worker-01/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-01-19 13:45:18 UTC] USER=www-data EUID=0 PID=1388427 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_user_sau_main_dev_db already exists
[2026-01-19 13:45:18 UTC] USER=www-data EUID=0 PID=1388454 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d fastorder_user_sau_main_dev_db --no-psqlrc
ERROR:  syntax error at or near "user"
LINE 1: CREATE SCHEMA IF NOT EXISTS user;
                                    ^
GRANT
ERROR:  syntax error at or near "user"
LINE 1: GRANT USAGE ON SCHEMA user TO debezium_user;
                              ^
ERROR:  syntax error at or near "user"
LINE 1: GRANT SELECT ON ALL TABLES IN SCHEMA user TO debezium_user;
                                             ^
ERROR:  syntax error at or near "user"
LINE 1: GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA user TO debez...
                                                       ^
ERROR:  syntax error at or near "user"
LINE 1: ALTER DEFAULT PRIVILEGES IN SCHEMA user GRANT SELECT ON TABL...
                                           ^
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
🔐 Creating replicator role for worker-01...
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : user-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SSL DIR     : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
  DNS → 10.100.1.232
  CA         : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Creating role: replicator with password
SET
CREATE ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/user/sau/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🎉 Done.
✅ Replicator role created for worker-01

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (user) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=user
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/user/run.sh
🔍 DEBUG_CHECKPOINT_03: No specific folder for user, using default
[DEBUG] Tracking substep start: steps/01-install/steps/default (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 🔸 Service: user (using default contracts schema)
🔍 DEBUG_CHECKPOINT_04: Executing default: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/run.sh
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting default contracts schema provisioning for SERVICE=user
[INFO] Environment: user-sau-main-dev
[INFO] Schema: user (contracts tables)
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Skipping Schema Setup on worker-01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

ℹ️  Schema setup only runs on coordinator
ℹ️  This is a worker-01 node - schemas replicate automatically

✅ Nothing to do on this node

✓ ✅ Worker worker-01 setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up standby replicas (1 per worker)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up standby: worker-01-standby-01 (replica of worker-01)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-19 13:45:25 UTC] USER=unknown EUID=33 PID=1388779 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-19 13:45:25 UTC] USER=unknown EUID=33 PID=1388786 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-19 13:45:25 UTC] USER=unknown EUID=33 PID=1388793 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-19 13:45:25 UTC] USER=unknown EUID=33 PID=1388800 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-19 13:45:25 UTC] USER=unknown EUID=33 PID=1388807 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-19 13:45:25 UTC] USER=unknown EUID=33 PID=1388815 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for user in sau-dev...
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566 JOB_UUID=fab5af05-2323-41ec-af35-3d55b2e9338d

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] PostgreSQL IP: 10.100.1.233
[INFO] Primary hostname: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01-standby-01...
[INFO]   db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.233

[INFO]   ✅ db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.233    db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  user-sau-main-dev.fastorder.com
  Alt CN:      user-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-19 13:45:28 UTC] USER=www-data EUID=0 PID=1388968 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:28 UTC] USER=www-data EUID=0 PID=1388977 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-19 13:45:28 UTC] USER=www-data EUID=0 PID=1388987 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1388932
[2026-01-19 13:45:28 UTC] USER=www-data EUID=0 PID=1388997 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1388932/ra_root.crt
[2026-01-19 13:45:28 UTC] USER=www-data EUID=0 PID=1389006 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1388932/ra_root.key
[2026-01-19 13:45:28 UTC] USER=www-data EUID=0 PID=1389015 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1388932/ra_root.crt
[2026-01-19 13:45:28 UTC] USER=www-data EUID=0 PID=1389024 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1388932/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = user-sau-main-dev.fastorder.com
[2026-01-19 13:45:30 UTC] USER=www-data EUID=0 PID=1389078 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1388932/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
[2026-01-19 13:45:30 UTC] USER=www-data EUID=0 PID=1389098 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1388932/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-19 13:45:30 UTC] USER=www-data EUID=0 PID=1389107 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-19 13:45:30 UTC] USER=www-data EUID=0 PID=1389116 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1388932/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:30 UTC] USER=www-data EUID=0 PID=1389125 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:30 UTC] USER=www-data EUID=0 PID=1389134 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:30 UTC] USER=www-data EUID=0 PID=1389143 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-19 13:45:30 UTC] USER=www-data EUID=0 PID=1389156 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
[2026-01-19 13:45:30 UTC] USER=www-data EUID=0 PID=1389165 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
[2026-01-19 13:45:30 UTC] USER=www-data EUID=0 PID=1389174 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-19 13:45:30 UTC] USER=www-data EUID=0 PID=1389183 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-19 13:45:30 UTC] USER=www-data EUID=0 PID=1389192 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:30 UTC] USER=www-data EUID=0 PID=1389201 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = user-sau-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:user-sau-main-dev.fastorder.com, DNS:user-sau-main-dev.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: user-sau-main-dev
Node:        worker-01-standby-01
Primary CN:  user-sau-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=user-sau-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389274 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389284 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389293 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389302 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389311 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389326 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389335 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389344 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389353 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389362 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389371 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389380 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389391 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389401 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389410 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389419 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389428 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389439 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389448 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389457 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389466 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389475 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389484 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:31 UTC] USER=www-data EUID=0 PID=1389510 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389519 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389528 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389537 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389546 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389555 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389564 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389574 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389584 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389593 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389602 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389611 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389621 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389632 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389642 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389659 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389668 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389677 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389686 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389695 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389704 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389713 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389722 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389731 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389741 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389751 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389761 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389770 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389779 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389788 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389797 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389806 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389815 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389824 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389833 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389842 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389851 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389860 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389870 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389880 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389889 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:32 UTC] USER=www-data EUID=0 PID=1389898 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1389907 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1389916 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1389925 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1389934 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1389943 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1389954 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1389964 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1389973 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1389982 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1389992 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1390005 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1390016 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1390025 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1390034 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1390043 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1390052 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1390061 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1390070 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1390079 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1390124 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1390133 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1390142 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1390151 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-19 13:45:33 UTC] USER=www-data EUID=0 PID=1390160 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390176 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390193 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390204 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390213 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390222 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390231 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390240 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390249 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390259 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390269 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390278 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390287 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390296 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390305 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390314 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390323 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390332 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390341 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390350 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390359 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390368 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390394 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390403 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390412 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390421 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390433 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390442 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390451 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390460 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390469 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390478 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390487 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390496 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390506 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:34 UTC] USER=www-data EUID=0 PID=1390516 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390525 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390534 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390543 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390552 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390561 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390570 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390579 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390588 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390597 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390606 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390615 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390625 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390635 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390644 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390654 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390664 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390681 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390698 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390711 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390720 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390729 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390739 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390759 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390779 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390797 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390818 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390828 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390837 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390847 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390856 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390867 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390880 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390899 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390908 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390917 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:45:35 UTC] USER=www-data EUID=0 PID=1390936 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:45:36 UTC] USER=www-data EUID=0 PID=1390946 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:45:36 UTC] USER=www-data EUID=0 PID=1390956 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:45:36 UTC] USER=www-data EUID=0 PID=1390965 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:45:36 UTC] USER=www-data EUID=0 PID=1390974 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:45:36 UTC] USER=www-data EUID=0 PID=1390983 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:45:36 UTC] USER=www-data EUID=0 PID=1390992 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:45:36 UTC] USER=www-data EUID=0 PID=1391001 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:45:36 UTC] USER=www-data EUID=0 PID=1391010 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:45:36 UTC] USER=www-data EUID=0 PID=1391020 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:45:36 UTC] USER=www-data EUID=0 PID=1391032 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:45:36 UTC] USER=www-data EUID=0 PID=1391073 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-19 13:45:36 UTC] USER=www-data EUID=0 PID=1391084 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-19 13:45:36 UTC] USER=www-data EUID=0 PID=1391094 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-19 13:45:36 UTC] USER=www-data EUID=0 PID=1391104 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-19 13:45:36 UTC] USER=www-data EUID=0 PID=1391113 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391149 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391164 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391174 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391183 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391192 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391201 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391210 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391220 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391229 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391238 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391247 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391256 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391265 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391275 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391300 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391317 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391326 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391337 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391347 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391356 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391388 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391397 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391418 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391428 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:45:37 UTC] USER=www-data EUID=0 PID=1391437 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391446 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391464 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391473 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391482 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391491 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391502 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391514 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391523 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391533 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391542 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391564 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391576 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391590 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391611 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391620 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391629 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391638 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391648 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391658 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391667 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391676 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391685 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391694 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391703 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391712 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391721 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391730 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391739 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391748 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391757 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391767 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391777 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:45:38 UTC] USER=www-data EUID=0 PID=1391786 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391795 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391804 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391813 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391822 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391831 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391840 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391849 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391858 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391867 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391876 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391886 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391896 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391905 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391914 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391923 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391932 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391941 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391951 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391960 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:45:39 UTC] USER=www-data EUID=0 PID=1391969 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-standby-01-postgresql environment: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com (10.100.1.233)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.233
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] Data dir:   /data/postgresql/17/user-sau-main-dev/worker-01-standby-01
[INFO] Port:       5432
[INFO] Hostname:   db-user-sau-main-dev-postgresql-worker-01-standby-01
[2026-01-19 13:45:40 UTC] USER=www-data EUID=0 PID=1392084 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:40 UTC] USER=www-data EUID=0 PID=1392105 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:41 UTC] USER=www-data EUID=0 PID=1392126 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:41 UTC] USER=www-data EUID=0 PID=1392147 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  user-sau-main-dev.fastorder.com
  Alt CN:      user-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-19 13:45:41 UTC] USER=www-data EUID=0 PID=1392196 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:41 UTC] USER=www-data EUID=0 PID=1392205 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-19 13:45:41 UTC] USER=www-data EUID=0 PID=1392215 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1392154
[2026-01-19 13:45:41 UTC] USER=www-data EUID=0 PID=1392225 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1392154/ra_root.crt
[2026-01-19 13:45:41 UTC] USER=www-data EUID=0 PID=1392235 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1392154/ra_root.key
[2026-01-19 13:45:41 UTC] USER=www-data EUID=0 PID=1392252 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1392154/ra_root.crt
[2026-01-19 13:45:41 UTC] USER=www-data EUID=0 PID=1392261 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1392154/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = user-sau-main-dev.fastorder.com
[2026-01-19 13:45:42 UTC] USER=www-data EUID=0 PID=1392307 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1392154/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
[2026-01-19 13:45:42 UTC] USER=www-data EUID=0 PID=1392316 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1392154/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-19 13:45:42 UTC] USER=www-data EUID=0 PID=1392325 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-19 13:45:42 UTC] USER=www-data EUID=0 PID=1392334 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1392154/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:42 UTC] USER=www-data EUID=0 PID=1392343 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:42 UTC] USER=www-data EUID=0 PID=1392352 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:45:43 UTC] USER=www-data EUID=0 PID=1392361 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-19 13:45:43 UTC] USER=www-data EUID=0 PID=1392372 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
[2026-01-19 13:45:43 UTC] USER=www-data EUID=0 PID=1392381 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
[2026-01-19 13:45:43 UTC] USER=www-data EUID=0 PID=1392390 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-19 13:45:43 UTC] USER=www-data EUID=0 PID=1392399 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-19 13:45:43 UTC] USER=www-data EUID=0 PID=1392408 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:45:43 UTC] USER=www-data EUID=0 PID=1392417 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = user-sau-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:user-sau-main-dev.fastorder.com, DNS:user-sau-main-dev.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: user-sau-main-dev
Node:        worker-01-standby-01
Primary CN:  user-sau-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=user-sau-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-19 13:45:43 UTC] USER=www-data EUID=0 PID=1392447 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-19 13:45:43 UTC] USER=www-data EUID=0 PID=1392456 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
[2026-01-19 13:45:43 UTC] USER=www-data EUID=0 PID=1392466 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-19 13:45:43 UTC] USER=www-data EUID=0 PID=1392514 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-worker-01-standby-01.service
[2026-01-19 13:45:43 UTC] USER=www-data EUID=0 PID=1392550 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-user-sau-main-dev-worker-01-standby-01
[OK]   No conflicting Postgres left on port 5432
[OK]   Generated new postgres password for initdb
[2026-01-19 13:46:06 UTC] USER=www-data EUID=0 PID=1393332 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.f0s1e6
[2026-01-19 13:46:06 UTC] USER=www-data EUID=0 PID=1393390 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/user-sau-main-dev
[2026-01-19 13:46:07 UTC] USER=www-data EUID=0 PID=1393416 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev
[2026-01-19 13:46:07 UTC] USER=www-data EUID=0 PID=1393439 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/user-sau-main-dev
[INFO] This is a standby. Using pg_basebackup from primary (worker-01)...
[INFO] Setting up replicator role and slot on primary (worker-01)...
ℹ️  Scanning primary for stuck queries from previous failed attempts...
ℹ️  Scanning for stuck queries (timeout: 30s)...
ℹ️  No stuck queries found
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : user-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SLOT        : worker_01_standby_01
  SSL DIR     : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
  DNS → 10.100.1.232
  CA         : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Role replicator already exists, updating password and ensuring REPLICATION privilege
SET
ALTER ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/user/sau/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🔧 Ensuring replication slot: worker_01_standby_01…
🆕 Creating replication slot worker_01_standby_01
SET
 pg_create_physical_replication_slot 
-------------------------------------
 (worker_01_standby_01,)
(1 row)

✅ Replication slot worker_01_standby_01 created.
🎉 Done.
[OK]   Replicator role and slot created on primary
[INFO] Creating replicator client certificates for connecting to primary (worker-01)...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:46:10 UTC] USER=www-data EUID=0 PID=1393626 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-19 13:46:10 UTC] USER=www-data EUID=0 PID=1393635 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-19 13:46:10 UTC] USER=www-data EUID=0 PID=1393644 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-19 13:46:10 UTC] USER=www-data EUID=0 PID=1393653 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-19 13:46:10 UTC] USER=www-data EUID=0 PID=1393662 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:46:10 UTC] USER=www-data EUID=0 PID=1393679 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:46:10 UTC] USER=www-data EUID=0 PID=1393688 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:46:10 UTC] USER=www-data EUID=0 PID=1393698 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:46:10 UTC] USER=www-data EUID=0 PID=1393710 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:46:10 UTC] USER=www-data EUID=0 PID=1393725 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:46:10 UTC] USER=www-data EUID=0 PID=1393735 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:46:10 UTC] USER=www-data EUID=0 PID=1393744 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:46:10 UTC] USER=www-data EUID=0 PID=1393753 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:46:10 UTC] USER=www-data EUID=0 PID=1393762 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:46:10 UTC] USER=www-data EUID=0 PID=1393771 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393780 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393789 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393798 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393807 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393816 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393825 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393834 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393843 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393852 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393861 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393870 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393896 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393905 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393916 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393925 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393934 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393943 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393952 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393961 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393979 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393988 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1393997 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394007 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394017 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394032 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394046 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394055 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394064 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394073 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394084 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394093 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394103 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394113 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394123 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394132 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394142 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394152 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394161 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394175 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394186 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394195 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394204 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:46:11 UTC] USER=www-data EUID=0 PID=1394213 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394222 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394231 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394240 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394249 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394258 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394268 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394278 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394287 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394296 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394305 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394314 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394323 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394332 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394341 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394350 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394359 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394368 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394377 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394387 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394397 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394406 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394415 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394424 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394433 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394442 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394451 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394460 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394469 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

[OK]   Replicator certificate created for worker-01 in /home/postgres/
[INFO] Using replicator certificates from primary worker-01...
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394497 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394519 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:46:12 UTC] USER=www-data EUID=0 PID=1394541 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt
[OK]   Replicator certificates verified at /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[OK]   root.crt verified at /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[INFO] Updating primary pg_hba.conf to allow replication...
[INFO]   Standby IP: 10.100.1.233/32 (standby's source IP)
[INFO]   Primary application IP: 10.100.1.232/32 (for local pg_basebackup)
[INFO]   Primary DNS IP: 10.100.1.232/32 (DNS resolution of db-user-sau-main-dev-postgresql-worker-01.fastorder.com)
[2026-01-19 13:46:13 UTC] USER=www-data EUID=0 PID=1394576 ACTION=passthru ARGS=grep -qxF # BEGIN standby-replication (managed) /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[2026-01-19 13:46:13 UTC] USER=www-data EUID=0 PID=1394630 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.233/32  scram-sha-256 
      $0==begin {inside=1}
      inside && $0==rule {found=1}
      $0==end {inside=0}
      END {exit found?0:1}
     /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[2026-01-19 13:46:13 UTC] USER=www-data EUID=0 PID=1394654 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl  replication  replicator  10.100.1.233/32  scram-sha-256 /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[2026-01-19 13:46:13 UTC] USER=www-data EUID=0 PID=1394677 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.232/32  scram-sha-256 
        $0==begin {inside=1}
        inside && $0==rule {found=1}
        $0==end {inside=0}
        END {exit found?0:1}
       /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[2026-01-19 13:46:13 UTC] USER=www-data EUID=0 PID=1394701 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl  replication  replicator  10.100.1.232/32  scram-sha-256 /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[INFO] Reloading primary PostgreSQL service...
[2026-01-19 13:46:13 UTC] USER=www-data EUID=0 PID=1394722 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-worker-01.service
[OK]   Primary pg_hba.conf updated and service reloaded
[WARN] Removing existing data directory: /data/postgresql/17/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:46:13 UTC] USER=www-data EUID=0 PID=1394745 ACTION=fsop ARGS=rm -rf /data/postgresql/17/user-sau-main-dev/worker-01-standby-01
[INFO] Primary host: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO] Using replicator cert: /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[INFO] Using replicator key: /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key (PKCS#8 format)
[INFO] Using CA cert: /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[INFO] Verifying postgres user can access certificates...
[ERR]  postgres user CANNOT read /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[INFO] File permissions:
lrwxrwxrwx 1 postgres ssl-cert 68 Jan 19 13:46 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt -> /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[INFO] Parent directory permissions:
drwx------ 2 postgres postgres 4096 Jan 19 13:46 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
drwx------ 6 postgres postgres 4096 Jan 18 23:43 /home/postgres/ssl/.postgresql/user-sau-main-dev
[WARN] Attempting to fix permissions (/usr/local/bin/fastorder-provisioning-wrapper.sh required)...
[INFO] Fixing /home/postgres/ directory...
[2026-01-19 13:46:14 UTC] USER=www-data EUID=0 PID=1394815 ACTION=fsop ARGS=chmod 755 /home/postgres/
[INFO] Fixing /home/postgres/ssl/.postgresql/...
[2026-01-19 13:46:14 UTC] USER=www-data EUID=0 PID=1394836 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/
[INFO] Fixing parent directory: /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:46:14 UTC] USER=www-data EUID=0 PID=1394859 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/user-sau-main-dev
[INFO] Fixing certificate directory: /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:46:14 UTC] USER=www-data EUID=0 PID=1394880 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[INFO] Fixing CA certificate: /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:46:14 UTC] USER=www-data EUID=0 PID=1394901 ACTION=fsop ARGS=chmod 644 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[OK]   Permissions fixed
[OK]   postgres user can now read /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt after permission fix
[2026-01-19 13:46:14 UTC] USER=www-data EUID=0 PID=1394922 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-user-sau-main-dev-worker-01-standby-01
[2026-01-19 13:46:14 UTC] USER=www-data EUID=0 PID=1394943 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-user-sau-main-dev-worker-01-standby-01
[2026-01-19 13:46:14 UTC] USER=www-data EUID=0 PID=1394964 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-user-sau-main-dev-worker-01-standby-01
[INFO] Checking primary database size before pg_basebackup...
[INFO] Total primary database size: 29 MB
[INFO] Estimated transfer time: ~0 minutes (at 10MB/s with compression)
[INFO] Retrieving replicator password from AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/replicator
[OK]   Replicator password retrieved successfully
[INFO] Starting pg_basebackup...
[2026-01-19 13:46:16 UTC] USER=www-data EUID=0 PID=1395058 ACTION=passthru ARGS=sudo -u postgres env PGPASSWORD=4fdUrcEKNirjtl6pfO2YEuBbBDxOb2hE PGSSLMODE=verify-full PGSSLCERT=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt PGSSLKEY=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key PGSSLROOTCERT=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /usr/lib/postgresql/17/bin/pg_basebackup -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -p 5432 -U replicator -D /data/postgresql/17/user-sau-main-dev/worker-01-standby-01 -Fp -Xs -P -R --checkpoint=fast --wal-method=stream --verbose
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/2000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1395068"
30526/30526 kB (100%), 0/1 tablespace (...-01-standby-01/global/pg_control)
30526/30526 kB (100%), 1/1 tablespace                                         
pg_basebackup: write-ahead log end point: 0/2000120
pg_basebackup: waiting for background process to finish streaming ...
pg_basebackup: syncing data to disk ...
pg_basebackup: renaming backup_manifest.tmp to backup_manifest
pg_basebackup: base backup completed
[OK]   pg_basebackup complete
[INFO] Fixing postgresql.auto.conf to use IP-based primary_conninfo (matching golden backup)...
[2026-01-19 13:46:17 UTC] USER=www-data EUID=0 PID=1395102 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-19 13:46:17 UTC] USER=www-data EUID=0 PID=1395124 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-19 13:46:17 UTC] USER=www-data EUID=0 PID=1395145 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-19 13:46:17 UTC] USER=www-data EUID=0 PID=1395154 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/standby.signal
[OK]   standby.signal verified and permissions set
[INFO] Fixing postgresql.conf with standby-specific settings...
[WARN] postgresql.conf not found at /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/postgresql.conf
[INFO] Verifying postgresql.auto.conf...
[WARN] postgresql.auto.conf not found - pg_basebackup may have failed
[2026-01-19 13:46:17 UTC] USER=www-data EUID=0 PID=1395180 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.f0s1e6
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-19 13:46:17 UTC] USER=www-data EUID=0 PID=1395230 ACTION=fsop ARGS=cp /tmp/tmp.X7z73alafn /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/pg_hba.conf
[2026-01-19 13:46:18 UTC] USER=www-data EUID=0 PID=1395251 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/pg_hba.conf
[2026-01-19 13:46:18 UTC] USER=www-data EUID=0 PID=1395273 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@user-sau-main-dev-worker-01-standby-01.service
[2026-01-19 13:46:18 UTC] USER=www-data EUID=0 PID=1395299 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.aR718b /etc/systemd/system/postgresql@user-sau-main-dev-worker-01-standby-01.service
[2026-01-19 13:46:18 UTC] USER=www-data EUID=0 PID=1395320 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@user-sau-main-dev-worker-01-standby-01.service
[OK]   systemd unit written
[2026-01-19 13:46:18 UTC] USER=www-data EUID=0 PID=1395342 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-19 13:46:18 UTC] USER=www-data EUID=0 PID=1395388 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-19 13:46:18 UTC] USER=www-data EUID=0 PID=1395425 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-19 13:46:19 UTC] USER=www-data EUID=0 PID=1395556 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-worker-01-standby-01.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-19 13:46:20 UTC] USER=www-data EUID=0 PID=1395605 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-worker-01-standby-01.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Configuring synchronous replication on primary worker-01...
[INFO] Current synchronous_standby_names: ''
[INFO] Initializing synchronous_standby_names with first standby
[INFO] New synchronous_standby_names: 'ANY 1 (worker_01_standby_01)'
[2026-01-19 13:46:20 UTC] USER=www-data EUID=0 PID=1395673 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_commit = on;
ALTER SYSTEM
[2026-01-19 13:46:20 UTC] USER=www-data EUID=0 PID=1395702 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_standby_names = 'ANY 1 (worker_01_standby_01)';
ALTER SYSTEM
[2026-01-19 13:46:20 UTC] USER=www-data EUID=0 PID=1395727 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
[OK]   ✅ Synchronous replication configured on primary
[OK]      Setting: ANY 1 (worker_01_standby_01)
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Skipping database/role provisioning on standby node (read-only)
[INFO]   Database/roles will be replicated from primary: worker-01
[INFO] Applying connection and memory optimizations...
[INFO] Standby will use primary's max_connections: 100
[INFO] Current settings: max_connections=100, work_mem=8MB
[INFO] Target settings (standby): max_connections=100, work_mem=8MB
[OK]   Connection settings already optimized
[INFO] Skipping password setting - this is a standby (read-only)
[INFO] Use primary's postgres password to connect to this standby
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] PostgreSQL IP: 10.100.1.233
[INFO] Primary hostname: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01-standby-01...
[INFO]   db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.233

[INFO]   ✅ db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.233    db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com


[OK]   PostgreSQL 'user-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key \
        host=db-user-sau-main-dev-postgresql-worker-01-standby-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        user-sau-main-dev-postgresql-worker-01-standby-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.233
[INFO]   Port:              5432
[INFO]   FQDN:              db-user-sau-main-dev-postgresql-worker-01-standby-01
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 6b53354f-af0c-46ce-9112-1ad9eae0ff4a
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 📦 03 role...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-19 13:46:26 UTC] USER=www-data EUID=0 PID=1396255 ACTION=fsop ARGS=test -f /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/standby.signal
⚠ This is a PostgreSQL STANDBY (read-only replica)
⚠ Skipping role creation - standby gets roles from primary via replication
⚠ Use the PRIMARY's credentials to connect to this standby


[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (user) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=user
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/user/run.sh
🔍 DEBUG_CHECKPOINT_03: No specific folder for user, using default
[DEBUG] Tracking substep start: steps/01-install/steps/default (RUN_UUID=702313ad-fe36-4738-baca-5168ec5e8566)
[INFO] 🔸 Service: user (using default contracts schema)
🔍 DEBUG_CHECKPOINT_04: Executing default: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/run.sh
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting default contracts schema provisioning for SERVICE=user
[INFO] Environment: user-sau-main-dev
[INFO] Schema: user (contracts tables)
[INFO] Identifier: worker-01-standby-01
[INFO] VM IP: 142.93.238.16

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Skipping Schema Setup on worker-01-standby-01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

ℹ️  Schema setup only runs on coordinator
ℹ️  This is a worker-01-standby-01 node - schemas replicate automatically

✅ Nothing to do on this node

✓ ✅ Standby worker-01-standby-01 setup completed

✓ ✅ PostgreSQL installation completed
[INFO] Discovering additional setup steps...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 02-pg-bouncer.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up PgBouncer connection pooling...
[2026-01-19 13:46:31 UTC] USER=www-data EUID=0 PID=1396461 ACTION=fsop ARGS=rm -f /tmp/pgbouncer-ip.service /tmp/pgbouncer.service
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] Checking for existing PgBouncer application environment in topology …
[OK]   Using existing PgBouncer environment:
[INFO]   IP:     10.100.1.184
[INFO]   FQDN:   db-user-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Domain: db-user-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO] Ensuring /etc/hosts entry for db-user-sau-main-dev-postgresql-bouncer.fastorder.com …
[OK]   /etc/hosts already contains entry for db-user-sau-main-dev-postgresql-bouncer.fastorder.com
[WARN] IP 10.100.1.184 is assigned to multiple interfaces:
    inet 10.100.1.217/32 scope global lo
       valid_lft forever preferred_lft forever
    inet 10.100.1.184/32 scope global lo
--
    inet 10.100.1.219/32 scope global eth0:219
       valid_lft forever preferred_lft forever
    inet 10.100.1.184/32 scope global eth0
[WARN] This may cause routing issues
[INFO] Final verification of /etc/hosts entry for db-user-sau-main-dev-postgresql-bouncer.fastorder.com …
[OK]   /etc/hosts correctly maps db-user-sau-main-dev-postgresql-bouncer.fastorder.com to 10.100.1.184
[WARN] IP 10.100.1.184 is already bound to other interface(s):
        inet 10.100.1.184/32 scope global lo
        inet 10.100.1.184/32 scope global eth0
[INFO] Attempting to also bind 10.100.1.184 to lo:pgbouncer ...
[2026-01-19 13:46:32 UTC] USER=www-data EUID=0 PID=1396572 ACTION=passthru ARGS=ip addr add 10.100.1.184/32 dev lo label lo:pgbouncer
RTNETLINK answers: File exists
[OK]   IP 10.100.1.184 is already bound to lo (may have different label)
[2026-01-19 13:46:32 UTC] USER=www-data EUID=0 PID=1396592 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-19 13:46:33 UTC] USER=www-data EUID=0 PID=1396687 ACTION=passthru ARGS=systemctl restart pgbouncer-ip@user-sau-main-dev.service
Job for pgbouncer-ip@user-sau-main-dev.service failed because the control process exited with error code.
See "systemctl status pgbouncer-ip@user-sau-main-dev.service" and "journalctl -xeu pgbouncer-ip@user-sau-main-dev.service" for details.
[2026-01-19 13:46:33 UTC] USER=www-data EUID=0 PID=1396697 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer-ip@user-sau-main-dev.service
[WARN] pgbouncer-ip@user-sau-main-dev.service is not active
[WARN] Check status: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@user-sau-main-dev.service
[2026-01-19 13:46:33 UTC] USER=www-data EUID=0 PID=1396721 ACTION=fsop ARGS=mkdir -p /etc/pgbouncer/user-sau-main-dev
[2026-01-19 13:46:34 UTC] USER=www-data EUID=0 PID=1396730 ACTION=fsop ARGS=mkdir -p /run/pgbouncer/user-sau-main-dev
[2026-01-19 13:46:34 UTC] USER=www-data EUID=0 PID=1396739 ACTION=fsop ARGS=mkdir -p /var/log/pgbouncer/user-sau-main-dev
[2026-01-19 13:46:34 UTC] USER=www-data EUID=0 PID=1396748 ACTION=fsop ARGS=chmod 750 /etc/pgbouncer/user-sau-main-dev
[2026-01-19 13:46:34 UTC] USER=www-data EUID=0 PID=1396757 ACTION=fsop ARGS=chmod 750 /run/pgbouncer/user-sau-main-dev
[2026-01-19 13:46:34 UTC] USER=www-data EUID=0 PID=1396766 ACTION=fsop ARGS=chmod 750 /var/log/pgbouncer/user-sau-main-dev
[2026-01-19 13:46:34 UTC] USER=www-data EUID=0 PID=1396779 ACTION=fsop ARGS=chown root:postgres /etc/pgbouncer/user-sau-main-dev
[2026-01-19 13:46:34 UTC] USER=www-data EUID=0 PID=1396791 ACTION=fsop ARGS=chown postgres:postgres /run/pgbouncer/user-sau-main-dev
[2026-01-19 13:46:34 UTC] USER=www-data EUID=0 PID=1396801 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbouncer/user-sau-main-dev
[INFO] Generating pgbouncer_admin client certificates...
[INFO] ⏳ This may take 30-60 seconds...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    pgbouncer_admin
Identifier:  pgbouncer
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        pgbouncer
  User (CN):   pgbouncer_admin
  Hostname:    db-user-sau-main-dev-postgresql-bouncer.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:46:34 UTC] USER=www-data EUID=0 PID=1396847 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-19 13:46:34 UTC] USER=www-data EUID=0 PID=1396856 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
[2026-01-19 13:46:34 UTC] USER=www-data EUID=0 PID=1396865 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-19 13:46:34 UTC] USER=www-data EUID=0 PID=1396874 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = pgbouncer_admin
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
[2026-01-19 13:46:35 UTC] USER=www-data EUID=0 PID=1396893 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
[2026-01-19 13:46:35 UTC] USER=www-data EUID=0 PID=1396905 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
[2026-01-19 13:46:35 UTC] USER=www-data EUID=0 PID=1396923 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-19 13:46:35 UTC] USER=www-data EUID=0 PID=1396932 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-19 13:46:35 UTC] USER=www-data EUID=0 PID=1396941 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt
[2026-01-19 13:46:35 UTC] USER=www-data EUID=0 PID=1396973 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-19 13:46:35 UTC] USER=www-data EUID=0 PID=1396990 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-19 13:46:35 UTC] USER=www-data EUID=0 PID=1396999 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-19 13:46:35 UTC] USER=www-data EUID=0 PID=1397008 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-19 13:46:35 UTC] USER=www-data EUID=0 PID=1397018 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-19 13:46:35 UTC] USER=www-data EUID=0 PID=1397027 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-19 13:46:35 UTC] USER=www-data EUID=0 PID=1397036 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt
[2026-01-19 13:46:35 UTC] USER=www-data EUID=0 PID=1397046 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
[2026-01-19 13:46:35 UTC] USER=www-data EUID=0 PID=1397056 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397065 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397074 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397088 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397102 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397144 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397153 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397189 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397216 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397225 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397234 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397243 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397253 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397263 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397281 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397292 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397302 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397311 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397320 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397329 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397347 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397356 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-19 13:46:36 UTC] USER=www-data EUID=0 PID=1397368 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397381 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397396 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397410 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397419 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397428 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397455 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397465 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397492 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397501 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397522 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397544 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397559 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397568 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397579 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397588 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397597 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397606 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397624 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397633 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397642 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397651 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397661 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397672 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:46:37 UTC] USER=www-data EUID=0 PID=1397681 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397694 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397712 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397721 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397730 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397739 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397748 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: pgbouncer_admin
Node: pgbouncer
FQDN: db-user-sau-main-dev-postgresql-bouncer.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-bouncer.fastorder.com -U pgbouncer_admin -d postgres

[OK]   mTLS client certificate present: /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[INFO] Creating symlinks to canonical certificates in /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend...
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397764 ACTION=fsop ARGS=mkdir -p /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397773 ACTION=fsop ARGS=mkdir -p /etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397782 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397791 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397800 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/root.crt
[INFO] Creating coordinator CA symlink for PostgreSQL server verification...
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397810 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] Verifying canonical certificate permissions...
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397819 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397828 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397837 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397846 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[OK]   Backend certificate symlinks created in /etc/ssl
[OK]   Coordinator CA symlink created for server verification
[OK]   Certificates already in canonical location - no symlinks needed
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397857 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/server.crt
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397866 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/server.key
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397876 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397886 ACTION=fsop ARGS=test -r /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] PgBouncer will use PostgreSQL coordinator CA: /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[OK]   PostgreSQL coordinator at db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432 is reachable
[INFO] Dumping SCRAM secrets from coordinator for PgBouncer auth_file …
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397913 ACTION=fsop ARGS=cp /tmp/tmp.A7R93N3KhL /etc/pgbouncer/user-sau-main-dev/userlist.txt
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397922 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/user-sau-main-dev/userlist.txt
[2026-01-19 13:46:38 UTC] USER=www-data EUID=0 PID=1397931 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/user-sau-main-dev/userlist.txt
[OK]   Auth file written: /etc/pgbouncer/user-sau-main-dev/userlist.txt
[INFO] Retrieved password from vault for pgbouncer_admin
[INFO] Ensuring PgBouncer admin role 'pgbouncer_admin' exists in Postgres (coordinator) …
[OK]   Role pgbouncer_admin created/updated successfully
[SECRETS] Setting credentials in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/pgbouncer_admin
✓ [SECRETS] Credentials updated in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/pgbouncer_admin
[INFO] ✅ PgBouncer admin password stored in centralized secrets vault
[INFO] Re-fetching SCRAM secrets after role creation to ensure pgbouncer_admin is included …
[2026-01-19 13:46:44 UTC] USER=www-data EUID=0 PID=1398053 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/user-sau-main-dev/userlist.txt
[2026-01-19 13:46:44 UTC] USER=www-data EUID=0 PID=1398062 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/user-sau-main-dev/userlist.txt
[OK]   Auth file updated with pgbouncer_admin SCRAM hash
[INFO] Auth file contains [2026-01-19 13:46:44 UTC] USER=www-data EUID=0 PID=1398080 ACTION=passthru ARGS=bash -c wc -l < '/etc/pgbouncer/user-sau-main-dev/userlist.txt'
4 user(s)
[OK]   Admin 'pgbouncer_admin' password generated and saved
[INFO] Configuring PostgreSQL to prevent Citus metadata sync hangs...
ALTER ROLE
[OK]   Disabled Citus metadata sync for pgbouncer_admin
[INFO] Verifying application database fastorder_user_sau_main_dev_db exists...
[OK]   ✓ Database fastorder_user_sau_main_dev_db exists
[INFO] Granting permissions to pgbouncer_admin on fastorder_user_sau_main_dev_db...
GRANT
[OK]   ✓ Granted CONNECT on fastorder_user_sau_main_dev_db to pgbouncer_admin
GRANT
[OK]   ✓ Granted USAGE on schema public to pgbouncer_admin
GRANT
[OK]   ✓ Granted SELECT on all tables to pgbouncer_admin
ALTER DATABASE
[OK]   Set synchronous_commit=local for fastorder_user_sau_main_dev_db
[INFO] Ensuring pg_hba.conf entry for pgbouncer_admin …
[INFO] Adding pg_hba.conf entries for pgbouncer_admin with cert auth …
[OK]   pg_hba.conf updated and PostgreSQL configuration reloaded
[2026-01-19 13:46:45 UTC] USER=unknown EUID=33 PID=1398122 ACTION=-u ARGS=postgres bash
ERROR: Invalid or unauthorized action: -u
[WARN] pg_hba.conf entry may not have loaded correctly
[INFO] Writing /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini …
[2026-01-19 13:46:46 UTC] USER=www-data EUID=0 PID=1398158 ACTION=fsop ARGS=cp /tmp/tmp.wrayCg5Nrc /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini
[2026-01-19 13:46:46 UTC] USER=www-data EUID=0 PID=1398171 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini
[2026-01-19 13:46:46 UTC] USER=www-data EUID=0 PID=1398192 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini
[2026-01-19 13:46:46 UTC] USER=www-data EUID=0 PID=1398212 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbouncer/user-sau-main-dev /run/pgbouncer/user-sau-main-dev /var/log/pgbouncer/user-sau-main-dev
[2026-01-19 13:46:46 UTC] USER=www-data EUID=0 PID=1398232 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/user-sau-main-dev/userlist.txt
[OK]   pgbouncer.ini ready
[INFO] Verifying TLS settings in pgbouncer.ini:
[2026-01-19 13:46:46 UTC] USER=www-data EUID=0 PID=1398245 ACTION=fsop ARGS=grep -E (client_tls_sslmode|server_tls) /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini
client_tls_sslmode = verify-full
server_tls_sslmode = verify-full
server_tls_ca_file = /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
server_tls_cert_file = /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
server_tls_key_file  = /etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[INFO] Verifying PgBouncer server certificate files:
[2026-01-19 13:46:46 UTC] USER=www-data EUID=0 PID=1398254 ACTION=fsop ARGS=test -r /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[OK]   Server cert readable by postgres: /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-01-19 13:46:46 UTC] USER=www-data EUID=0 PID=1398263 ACTION=fsop ARGS=test -r /etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[OK]   Server key readable by postgres: /etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[INFO] Verifying coordinator CA certificate:
[2026-01-19 13:46:46 UTC] USER=www-data EUID=0 PID=1398273 ACTION=fsop ARGS=test -r /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[OK]   Coordinator CA readable by postgres: /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] Preflight: stopping any conflicting PgBouncer on 6432 …
[2026-01-19 13:46:47 UTC] USER=www-data EUID=0 PID=1398283 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer.service
[2026-01-19 13:46:47 UTC] USER=www-data EUID=0 PID=1398293 ACTION=passthru ARGS=systemctl stop pgbouncer@user-sau-main-dev.service
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json?all=1": dial unix /var/run/docker.sock: connect: permission denied
[WARN] Killing existing pgbouncer processes: 1345629
[2026-01-19 13:48:17 UTC] USER=www-data EUID=0 PID=1400631 ACTION=passthru ARGS=bash -c kill -9 1345629
[2026-01-19 13:48:19 UTC] USER=www-data EUID=0 PID=1400735 ACTION=passthru ARGS=systemctl daemon-reload
[OK]   systemd unit installed: pgbouncer@user-sau-main-dev.service
[INFO] Running pre-flight IP conflict check for 10.100.1.184:6432 …
[WARN] IP conflict checker not found at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/check-ip-conflicts.sh
[WARN] Skipping pre-flight check - conflicts may occur
[INFO] Starting PgBouncer (user-sau-main-dev) …
[2026-01-19 13:48:20 UTC] USER=www-data EUID=0 PID=1400871 ACTION=passthru ARGS=systemctl restart pgbouncer@user-sau-main-dev.service
[2026-01-19 13:48:20 UTC] USER=www-data EUID=0 PID=1400882 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer@user-sau-main-dev.service
[OK]   Service ACTIVE
[INFO] Verifying auth_file before probing …
[INFO] Auth file contains 4 user(s)
[WARN] Auth file does NOT contain pgbouncer_admin entry - authentication will fail
[INFO] Probing admin console via SSL (psql to database 'pgbouncer') …
[INFO] Retrieved password from vault for admin console probe
[WARN] Admin console probe failed (see error below)
psql: error: connection to server at "10.100.1.184", port 6432 failed: server certificate for "db-user-sau-main-dev-postgresql-bouncer.fastorder.com" (and 6 other names) does not match host name "10.100.1.184"
[WARN] Troubleshooting:
[WARN]   1. Check auth_file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/user-sau-main-dev/userlist.txt
[WARN]   2. Test with: PGPASSWORD='yvonAdiGcvLlur+JNgqyr7ru' psql -h 10.100.1.184 -p 6432 -U pgbouncer_admin -d pgbouncer
[WARN]   3. Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@user-sau-main-dev.service -n 50

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Running Comprehensive PgBouncer Verification Tests
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Password extracted: yvonAdiGcv... (using postgres user certificates)

[INFO] Test 1/7: Admin Console - SHOW POOLS
 database  |   user    | cl_active | cl_waiting | cl_active_cancel_req | cl_waiting_cancel_req | sv_active | sv_active_cancel | sv_being_canceled | sv_idle | sv_used | sv_tested | sv_login | maxwait | maxwait_us | pool_mode | load_balance_hosts 
-----------+-----------+-----------+------------+----------------------+-----------------------+-----------+------------------+-------------------+---------+---------+-----------+----------+---------+------------+-----------+--------------------
 pgbouncer | pgbouncer |         2 |          0 |                    0 |                     0 |         0 |                0 |                 0 |       0 |       0 |         0 |        0 |       0 |          0 | statement | 
(1 row)

[OK]   ✓ SHOW POOLS: SUCCESS

[INFO] Test 2/7: Admin Console - SHOW VERSION
[OK]   ✓ SHOW VERSION: PgBouncer 1.24.1

[INFO] Test 3/7: Admin Console - SHOW STATS
 database  | total_server_assignment_count | total_xact_count | total_query_count | total_received | total_sent | total_xact_time | total_query_time | total_wait_time | total_client_parse_count | total_server_parse_count | total_bind_count | avg_server_assignment_count | avg_xact_count | avg_query_count | avg_recv | avg_sent | avg_xact_time | avg_query_time | avg_wait_time | avg_client_parse_count | avg_server_parse_count | avg_bind_count 
-----------+-------------------------------+------------------+-------------------+----------------+------------+-----------------+------------------+-----------------+--------------------------+--------------------------+------------------+-----------------------------+----------------+-----------------+----------+----------+---------------+----------------+---------------+------------------------+------------------------+----------------
 pgbouncer |                             0 |                4 |                 4 |              0 |          0 |               0 |                0 |               0 |                        0 |                        0 |                0 |                           0 |              0 |               0 |        0 |        0 |             0 |              0 |             0 |                      0 |                      0 |              0
(1 row)

[OK]   ✓ SHOW STATS: SUCCESS

[INFO] Test 4/7: Admin Console - SHOW DATABASES
              name              |                           host                            | port |            database            | force_user | pool_size | min_pool_size | reserve_pool_size | server_lifetime | pool_mode | load_balance_hosts | max_connections | current_connections | max_client_connections | current_client_connections | paused | disabled 
--------------------------------+-----------------------------------------------------------+------+--------------------------------+------------+-----------+---------------+-------------------+-----------------+-----------+--------------------+-----------------+---------------------+------------------------+----------------------------+--------+----------
 fastorder_user_sau_main_dev_db | db-user-sau-main-dev-postgresql-coordinator.fastorder.com | 5432 | fastorder_user_sau_main_dev_db |            |       100 |             0 |                20 |            3600 |           |                    |               0 |                   0 |                      0 |                          0 |      0 |        0
 pgbouncer                      |                                                           | 6432 | pgbouncer                      | pgbouncer  |         2 |             0 |                 0 |            3600 | statement |                    |               0 |                   0 |                      0 |                          2 |      0 |        0
(2 rows)

[OK]   ✓ SHOW DATABASES: SUCCESS

[INFO] Test 5/7: Admin Console - SHOW CONFIG
[OK]   ✓ SHOW CONFIG: SUCCESS
[INFO]   Key settings:
[INFO]     client_tls_sslmode = verify-full|disable|yes
[INFO]     max_client_conn = 2048|100|yes
[INFO]     pool_mode = transaction|session|yes
[INFO]     server_tls_sslmode = verify-full|prefer|yes
psql   "host=db-user-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_user_sau_main_dev_db user=pgbouncer_admin password=yvonAdiGcvLlur+JNgqyr7ru    connect_timeout=5 sslmode=verify-full    sslrootcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt    sslcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt    sslkey=/etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key"   --no-psqlrc -Atc 'SELECT version();'

[INFO] Test 6/7: Application Database - SELECT version()
[WARN] ✗ Application database query: FAILED (timeout or connection issue)
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[INFO] Test 7/8: Application Database - Connection Details
[WARN] ✗ Connection details: FAILED (timeout or connection issue)
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[INFO] Test 8/8: End-to-End Application Routing - Pool Verification
[INFO]   Running actual queries through PgBouncer to verify routing and pooling...
[WARN] ✗ End-to-end routing verification: FAILED - All 3 queries failed
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[WARN]    Otherwise check if database fastorder_user_sau_main_dev_db exists and user pgbouncer_admin has permissions

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Verification Complete - Tests 1-5 PASSED (Admin console verified)
[WARN]   Tests 6-8 FAILED - Application database not accessible
[WARN]   This is expected if Citus is not set up yet
[WARN]   Run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[OK]   PgBouncer is up for user-sau-main-dev

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Connection Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Password stored in: AWS Secrets Manager (fastorder/db/web/ksa/main/dev/postgresqluser/sau/main/dev/coordinator-pgbouncer_admin)
Current password: yvonAdiGcvLlur+JNgqyr7ru

1. Admin Console (using IP address to avoid DNS/SSL issues):
   psql "host=10.100.1.184 port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=yvonAdiGcvLlur+JNgqyr7ru sslmode=verify-full sslrootcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

2. Admin Console (using hostname):
   psql "host=db-user-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=yvonAdiGcvLlur+JNgqyr7ru sslmode=verify-full sslrootcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

3. Application Database:
   psql "host=db-user-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_user_sau_main_dev_db sslkey=/etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=yvonAdiGcvLlur+JNgqyr7ru sslmode=verify-full sslrootcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

4. Using .pgpass file:
   echo "db-user-sau-main-dev-postgresql-bouncer.fastorder.com:6432:*:pgbouncer_admin:yvonAdiGcvLlur+JNgqyr7ru" >> ~/.pgpass
   chmod 600 ~/.pgpass
   psql -h db-user-sau-main-dev-postgresql-bouncer.fastorder.com -p 6432 -U pgbouncer_admin -d fastorder_user_sau_main_dev_db

5. Retrieve password from vault:
   source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
   PGPASSWORD="$(get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password')" \
     psql -h 10.100.1.184 -p 6432 -U pgbouncer_admin -d pgbouncer -c "SHOW POOLS;"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Architecture
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  • Default db 'fastorder_user_sau_main_dev_db' → Citus coordinator (db-user-sau-main-dev-postgresql-coordinator.fastorder.com)
  • Worker access: 'fastorder_user_sau_main_dev_db_worker_1', 'fastorder_user_sau_main_dev_db_worker_2', … (if exist)
  • Client TLS: require (password auth) / verify-full (mTLS with certs)
  • Server TLS: verify-full (PgBouncer validates PostgreSQL certs)
  • Auth: SCRAM-SHA-256 via /etc/pgbouncer/user-sau-main-dev/userlist.txt
  • Pool mode: transaction (stateless connections)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Management
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Service Status:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer@user-sau-main-dev.service
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@user-sau-main-dev.service

Logs:
  command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@user-sau-main-dev.service -f
  /usr/local/bin/fastorder-provisioning-wrapper.sh tail -f /var/log/pgbouncer/user-sau-main-dev/pgbouncer.log

Reload Config:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@user-sau-main-dev.service

Restart:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@user-sau-main-dev.service

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Files
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Config:        /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini
Auth file:     /etc/pgbouncer/user-sau-main-dev/userlist.txt
Server cert:   /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/server.crt
Server key:    /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/server.key
CA cert:       /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt
PG CA:         /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
Logs:          /var/log/pgbouncer/user-sau-main-dev/pgbouncer.log

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Troubleshooting
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


If "SASL authentication failed":
  1. Check auth file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/user-sau-main-dev/userlist.txt
  2. Verify pgbouncer_admin is present with SCRAM hash
  3. Get password from vault:
     source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
     get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password'
  4. Reload PgBouncer: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@user-sau-main-dev.service

If "no pg_hba.conf entry":
  1. Check pg_hba.conf on coordinator
  2. Add rule: hostssl all pgbouncer_admin 10.100.1.184/32 cert clientcert=verify-full
  3. Reload PostgreSQL

To add users to PgBouncer:
  1. Create user in PostgreSQL with password
  2. Re-run SCRAM dump:
     psql "host=db-user-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 dbname=postgres user=postgres \
       sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt \
       sslcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt sslkey=/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key" \
       -Atc "SELECT '\"' || rolname || '\" \"' || rolpassword || '\"' \
             FROM pg_authid WHERE rolpassword LIKE 'SCRAM-SHA-256%' \
             AND rolcanlogin ORDER BY rolname;" | command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop tee /etc/pgbouncer/user-sau-main-dev/userlist.txt
  3. Reload: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@user-sau-main-dev.service

[INFO] Registering PgBouncer node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        user-sau-main-dev-pgbouncer
[INFO]   Identifier Parent: postgresql
[INFO]   IP:                10.100.1.184
[INFO]   Port:              6432
[INFO]   FQDN:              db-user-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: c866fe26-0c2d-4619-a98d-8cd82c922b78
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[OK]   PgBouncer node registered to observability API
✓ ✅ PgBouncer setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 03-citus-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] CITUS DISTRIBUTED CLUSTER SETUP
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Phase 1: Installing Citus extension on workers...
[INFO] Phase 2: Setting up coordinator and registering workers...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 📦 PHASE 1: Installing Citus extension on 1 worker(s)...

[INFO] → Worker 1/1: Installing Citus on worker-01...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS CLUSTER SETUP
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔧 Setting up Citus Worker...
[INFO] Temporarily disabling synchronous replication for extension installation...
t
[INFO] Installing Citus extension on worker...
[OK]   Citus extension installed on worker
[INFO] Restoring synchronous replication settings...
t
[INFO] Worker Citus extension installed - registration will happen when coordinator setup runs

[OK]   Citus setup complete for worker-01
[INFO] ═══════════════════════════════════════════════════════════════════════════════
✓   ✅ Citus extension installed on worker-01

✓ ✅ Phase 1 Complete: All 1 workers have Citus extension installed

[INFO] 🔧 PHASE 2: Setting up Citus coordinator and registering workers...

[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS CLUSTER SETUP
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔧 Setting up Citus Coordinator...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] DIAGNOSTIC: Configuration Variables
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PG_WORKERS_NUM: 1
[INFO] ENV_ID: user-sau-main-dev
[INFO] DOMAIN: fastorder.com
[INFO] PORT: 5432
[INFO] SOCKET_DIR: /var/run/postgresql-user-sau-main-dev-coordinator
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Ensuring postgres client certificates exist for coordinator...
[OK]   Postgres client certificates already exist for coordinator
[INFO] Adding citus_cert_map to coordinator pg_ident.conf...
[OK]   pg_ident.conf updated for coordinator
[INFO] Installing Citus extension on coordinator...
[OK]   Citus extension installed on coordinator (postgres database)
[INFO] Installing Citus extension on application database: fastorder_user_sau_main_dev_db...
[OK]   Citus extension installed on application database: fastorder_user_sau_main_dev_db
[INFO] Configuring Citus SSL connection parameters...
[2026-01-19 13:48:44 UTC] USER=www-data EUID=0 PID=1401669 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-coordinator.service
[OK]   ✅ Citus SSL connection parameters configured: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[WARN] Node not identified as coordinator, initializing...
[INFO] Checking coordinator configuration...
[INFO] Persisting citus.local_hostname to postgresql.conf...
[2026-01-19 13:48:47 UTC] USER=www-data EUID=0 PID=1401746 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/user-sau-main-dev/coordinator/postgresql.conf
[2026-01-19 13:48:47 UTC] USER=www-data EUID=0 PID=1401767 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-coordinator.service
[OK]   ✅ citus.local_hostname persisted to config and reloaded
[INFO] Configuring coordinator hostname in postgres database: db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432

[OK]   ✅ Coordinator hostname set to db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432 in postgres database
[INFO] Checking coordinator configuration in application database: fastorder_user_sau_main_dev_db...
[WARN] ⚠️  Coordinator registered as 'localhost' in application database, fixing...
[INFO] Configuring coordinator hostname in application database: db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432
[OK]   ✅ Coordinator hostname set to db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432 in application database
[INFO] Validating coordinator configuration before worker registration...
[OK]   ✅ Coordinator hostname validated: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[OK]   ✅ citus_tables view is accessible
[INFO] Checking coordinator self-registration...
[OK]   ✅ Coordinator is already self-registered
[INFO] Configuring coordinator shard placement policy...
[OK]   ✅ Coordinator already configured in postgres database (shouldhaveshards = false)
[WARN] ⚠️  Coordinator has 66 shards in fastorder_user_sau_main_dev_db - cannot set shouldhaveshards=false
[WARN]    You must rebalance shards to workers first, then run this setup again
[WARN]    Skipping shouldhaveshards configuration for application database
[INFO] Registering 1 worker(s) to Citus cluster...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PRE-FLIGHT: Checking worker availability...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking worker worker-01...
[INFO]   FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[OK]   ✅ Worker worker-01 is reachable via SSL
[OK]   All workers are reachable - proceeding with registration

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Adding Citus worker: db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Adding citus_cert_map to worker-01 pg_ident.conf...
[OK]   pg_ident.conf updated for worker-01
[INFO] Configuring worker worker-01 HBA for coordinator (10.100.1.231) access...
[OK]   Worker worker-01 HBA configured for coordinator (10.100.1.231)
[INFO] Adding replication rules for 3 standby(s)...
[OK]   Replication rules already exist for worker-01
[INFO] Reloading worker worker-01 to apply HBA changes...
[2026-01-19 13:48:50 UTC] USER=www-data EUID=0 PID=1401928 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-worker-01.service
[INFO] Configuring coordinator HBA for worker worker-01 (10.100.1.232) access...
[OK]   Coordinator HBA configured for worker worker-01 (10.100.1.232)
[INFO] Reloading coordinator to apply HBA changes...
[2026-01-19 13:48:50 UTC] USER=www-data EUID=0 PID=1401960 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-coordinator.service
[INFO] Ensuring postgres client certificates exist for worker-01...
[OK]   Postgres client certificates already exist for worker-01
[INFO] Configuring citus.node_conninfo on worker-01...
[2026-01-19 13:48:50 UTC] USER=www-data EUID=0 PID=1401976 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-worker-01.service
[OK]   citus.node_conninfo configured on worker-01
[INFO] Temporarily relaxing sync-rep on worker worker-01...
t
[OK]   Worker worker-01 sync-rep relaxed (was: sync_commit=on)
[INFO] Ensuring Citus extension on worker databases...
CREATE EXTENSION
CREATE EXTENSION
[INFO] Running citus_add_node with 180s timeout...
NOTICE:  shards are still on the coordinator after adding the new node
HINT:  Use SELECT rebalance_table_shards(); to balance shards data between workers and coordinator or SELECT citus_drain_node('db-user-sau-main-dev-postgresql-coordinator.fastorder.com',5432); to permanently move shards away from the coordinator.
2
[INFO] Restoring worker worker-01 sync-rep settings...
t
[OK]   Worker worker-01 sync-rep restored
[OK]   ✅ Worker db-user-sau-main-dev-postgresql-worker-01.fastorder.com successfully added to Citus cluster
[INFO]    Node ID: 2
[INFO]    Registered in: postgres, fastorder_user_sau_main_dev_db
[OK]   Worker worker-01 registration successful
[INFO] Configuring worker worker-01 shard placement policy...
[OK]   ✅ Worker worker-01 configured to hold shards in all databases


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] POST-REGISTRATION: Verifying cluster state...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Expected workers: 1
[INFO] Registered workers: 1
[OK]   ✅ All 1 workers successfully registered!

[INFO] Citus cluster configuration:
db-user-sau-main-dev-postgresql-coordinator.fastorder.com  5432  0  t  primary  f
db-user-sau-main-dev-postgresql-worker-01.fastorder.com    5432  1  t  primary  t

[INFO] Note: groupid=0 is the coordinator, groupid>0 are workers
[INFO]       shouldhaveshards: false=query router only, true=holds data shards

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] FINAL VALIDATION: Verifying configuration persistence...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:48:53 UTC] USER=www-data EUID=0 PID=1402201 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/user-sau-main-dev/coordinator/postgresql.conf
[OK]   ✅ citus.local_hostname persisted in postgresql.conf
[OK]   ✅ All 1 worker(s) successfully registered and verified

[OK]   ✅ All validation checks passed
[OK]   Citus coordinator setup complete

[OK]   Citus setup complete for coordinator
[INFO] ═══════════════════════════════════════════════════════════════════════════════

✓ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ CITUS CLUSTER SETUP COMPLETED SUCCESSFULLY
✓    Coordinator: Ready and accepting connections
✓    Workers registered: 1
✓ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 05-backup-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up coordinator backup...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for user-sau-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-01-19 13:48:55 UTC] USER=www-data EUID=0 PID=1402299 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/user-sau-main-dev
[2026-01-19 13:48:55 UTC] USER=www-data EUID=0 PID=1402310 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/user-sau-main-dev
[2026-01-19 13:48:55 UTC] USER=www-data EUID=0 PID=1402336 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-01-19 13:48:55 UTC] USER=www-data EUID=0 PID=1402345 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-19 13:48:55 UTC] USER=www-data EUID=0 PID=1402360 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-19 13:48:55 UTC] USER=www-data EUID=0 PID=1402370 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-19 13:48:58 UTC] USER=www-data EUID=0 PID=1402528 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-19 13:48:58 UTC] USER=www-data EUID=0 PID=1402559 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-19 13:48:58 UTC] USER=www-data EUID=0 PID=1402601 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-01-19 13:48:58 UTC] USER=www-data EUID=0 PID=1402626 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/user-sau-main-dev
[2026-01-19 13:48:58 UTC] USER=www-data EUID=0 PID=1402636 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/user-sau-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-user-sau-main-dev
[2026-01-19 13:48:59 UTC] USER=www-data EUID=0 PID=1402657 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-19 13:48:59 UTC] USER=www-data EUID=0 PID=1402666 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-01-19 13:48:59 UTC] USER=www-data EUID=0 PID=1402675 ACTION=fsop ARGS=find /data/postgresql/17/user-sau-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-01-19 13:48:59 UTC] USER=www-data EUID=0 PID=1402684 ACTION=fsop ARGS=chown -R postgres:postgres /data/postgresql/17/user-sau-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-19 13:48:59 UTC] USER=www-data EUID=0 PID=1402699 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-01-19 13:48:59 UTC] USER=www-data EUID=0 PID=1402711 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-01-19 13:48:59 UTC] USER=www-data EUID=0 PID=1402720 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-19 13:48:59 UTC] USER=www-data EUID=0 PID=1402729 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/user-sau-main-dev/coordinator/PG_VERSION
[2026-01-19 13:48:59 UTC] USER=www-data EUID=0 PID=1402739 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza user-sau-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-19 13:48:59 UTC] USER=www-data EUID=0 PID=1402804 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-19 13:49:01 UTC] USER=www-data EUID=0 PID=1402845 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-coordinator.service
[2026-01-19 13:49:05 UTC] USER=www-data EUID=0 PID=1402917 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-19 13:49:05 UTC] USER=www-data EUID=0 PID=1402952 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=user-sau-main-dev-coordinator --log-level-console=info check
2026-01-19 13:49:05.806 P00   INFO: check command begin 2.56.0: --exec-id=1402960-96500e8f --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/user-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-user-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/user-sau-main-dev --stanza=user-sau-main-dev-coordinator
2026-01-19 13:49:05.829 P00   INFO: check repo1 configuration (primary)
2026-01-19 13:49:05.843 P00  ERROR: [028]: backup and archive info files exist but do not match the database
                                    HINT: is this the correct stanza?
                                    HINT: did an error occur during stanza-upgrade?
2026-01-19 13:49:05.843 P00   INFO: check command end: aborted with exception [028]
[WARN] ⚠️  Stanza verification failed - this may be normal if WAL archiving hasn't started yet
[WARN]    The backup system is configured and will work once WAL segments are generated

[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-19 13:49:05 UTC] USER=www-data EUID=0 PID=1402973 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|user-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-user-sau-main-dev.sh
[2026-01-19 13:49:06 UTC] USER=www-data EUID=0 PID=1403008 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|user-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-user-sau-main-dev.sh
[2026-01-19 13:49:06 UTC] USER=www-data EUID=0 PID=1403017 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-user-sau-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-19 13:49:06 UTC] USER=www-data EUID=0 PID=1403035 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-user-sau-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-01-19 13:49:06 UTC] USER=www-data EUID=0 PID=1403063 ACTION=fsop ARGS=sed -i s|__ENV_ID__|user-sau-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-19 13:49:06 UTC] USER=www-data EUID=0 PID=1403072 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/data/postgresql/17/user-sau-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-19 13:49:06 UTC] USER=www-data EUID=0 PID=1403090 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-19 13:49:06.352 P00   INFO: start command begin 2.56.0: --exec-id=1403111-d7f67153 --log-level-console=info --log-level-file=debug --stanza=user-sau-main-dev-coordinator
2026-01-19 13:49:06.358 P00   WARN: stop file does not exist for stanza user-sau-main-dev-coordinator
2026-01-19 13:49:06.358 P00   INFO: start command end: completed successfully (9ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-19 13:49:06.416 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=1403122-173bf50a --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/user-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-user-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/user-sau-main-dev --stanza=user-sau-main-dev-coordinator
2026-01-19 13:49:06.418 P00   INFO: stanza-upgrade for stanza 'user-sau-main-dev-coordinator' on repo1
2026-01-19 13:49:06.430 P00   INFO: stanza-upgrade command end: completed successfully (17ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-19 13:49:06 UTC] USER=www-data EUID=0 PID=1403144 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260119-134906.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-01-19 13:49:15 UTC] USER=www-data EUID=0 PID=1403385 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-1402246.log /var/log/pgbackrest/initial-backup-20260119-134906.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260119-134906.log
   2026-01-19 13:49:15.072 P00   INFO: repo1: remove expired backup 20260118-234609F
   2026-01-19 13:49:15.115 P00   INFO: repo1: 17-23 remove archive, start = 000000010000000000000003, stop = 000000010000000000000005
   2026-01-19 13:49:15.115 P00   INFO: repo1: 17-24 no archive to remove
   2026-01-19 13:49:15.116 P00   INFO: repo1: 17-25 remove archive, start = 000000010000000000000002, stop = 000000010000000000000002
   2026-01-19 13:49:15.116 P00   INFO: expire command end: completed successfully (54ms)

[INFO] Current backups:
stanza: user-sau-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000006/00000001000000040000006E

        full backup: 20260118-234628F
            timestamp start/stop: 2026-01-18 23:46:28+00 / 2026-01-18 23:46:34+00
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.6MB, backup size: 5.6MB

        diff backup: 20260118-234628F_20260119-020006D
            timestamp start/stop: 2026-01-19 02:00:06+00 / 2026-01-19 02:00:15+00
            wal start/stop: 0000000100000000000000BF / 0000000100000000000000C9
            database size: 37.7MB, database backup size: 9.2MB
            repo1: backup set size: 5.7MB, backup size: 1.8MB
            backup reference total: 1 full

    db (prior)
        wal archive min/max (17): 000000010000000000000003/00000001000000000000000A

        full backup: 20260119-133359F
            timestamp start/stop: 2026-01-19 13:33:59+00 / 2026-01-19 13:34:07+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.6MB, backup size: 5.6MB

        full backup: 20260119-133418F
            timestamp start/stop: 2026-01-19 13:34:18+00 / 2026-01-19 13:34:20+00
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.6MB, backup size: 5.6MB

    db (current)
        wal archive min/max (17): 000000010000000000000003/000000010000000000000003

        full backup: 20260119-134906F
            timestamp start/stop: 2026-01-19 13:49:06+00 / 2026-01-19 13:49:14+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.6MB, backup size: 5.6MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         user-sau-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/user-sau-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/user-sau-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=user-sau-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up worker backups for 1 worker(s)...
[INFO] Setting up backup for: worker-01
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for user-sau-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-01-19 13:49:15 UTC] USER=www-data EUID=0 PID=1403452 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/user-sau-main-dev
[2026-01-19 13:49:15 UTC] USER=www-data EUID=0 PID=1403461 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/user-sau-main-dev
[2026-01-19 13:49:15 UTC] USER=www-data EUID=0 PID=1403470 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-01-19 13:49:15 UTC] USER=www-data EUID=0 PID=1403479 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-19 13:49:15 UTC] USER=www-data EUID=0 PID=1403488 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-19 13:49:15 UTC] USER=www-data EUID=0 PID=1403497 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-19 13:49:17 UTC] USER=www-data EUID=0 PID=1403526 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-19 13:49:17 UTC] USER=www-data EUID=0 PID=1403535 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-19 13:49:17 UTC] USER=www-data EUID=0 PID=1403545 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-01-19 13:49:17 UTC] USER=www-data EUID=0 PID=1403554 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/user-sau-main-dev
[2026-01-19 13:49:17 UTC] USER=www-data EUID=0 PID=1403564 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/user-sau-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-user-sau-main-dev
[2026-01-19 13:49:17 UTC] USER=www-data EUID=0 PID=1403585 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-19 13:49:17 UTC] USER=www-data EUID=0 PID=1403594 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-01-19 13:49:17 UTC] USER=www-data EUID=0 PID=1403603 ACTION=fsop ARGS=find /data/postgresql/17/user-sau-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-01-19 13:49:17 UTC] USER=www-data EUID=0 PID=1403612 ACTION=fsop ARGS=chown -R postgres:postgres /data/postgresql/17/user-sau-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-19 13:49:17 UTC] USER=www-data EUID=0 PID=1403630 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-01-19 13:49:18 UTC] USER=www-data EUID=0 PID=1403639 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-19 13:49:18 UTC] USER=www-data EUID=0 PID=1403660 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/user-sau-main-dev/coordinator/PG_VERSION
[2026-01-19 13:49:18 UTC] USER=www-data EUID=0 PID=1403670 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza user-sau-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-19 13:49:18 UTC] USER=www-data EUID=0 PID=1403752 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-19 13:49:21 UTC] USER=www-data EUID=0 PID=1403807 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-coordinator.service
[2026-01-19 13:49:24 UTC] USER=www-data EUID=0 PID=1403942 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-19 13:49:24 UTC] USER=www-data EUID=0 PID=1403975 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=user-sau-main-dev-coordinator --log-level-console=info check
2026-01-19 13:49:24.824 P00   INFO: check command begin 2.56.0: --exec-id=1403982-b86059aa --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/user-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-user-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/user-sau-main-dev --stanza=user-sau-main-dev-coordinator
2026-01-19 13:49:24.842 P00   INFO: check repo1 configuration (primary)
2026-01-19 13:49:24.896 P00   INFO: check repo1 archive for WAL (primary)
2026-01-19 13:49:25.198 P00   INFO: WAL segment 000000010000000000000005 successfully archived to '/var/lib/pgbackrest/backup/user-sau-main-dev/archive/user-sau-main-dev-coordinator/17-25/0000000100000000/000000010000000000000005-f4d75d575ed2fc8b73162b58d0e97f98e47b310f.lz4' on repo1
2026-01-19 13:49:25.198 P00   INFO: check command end: completed successfully (379ms)
[INFO] ✅ Stanza verification passed

[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-19 13:49:25 UTC] USER=www-data EUID=0 PID=1404006 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|user-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-user-sau-main-dev.sh
[2026-01-19 13:49:25 UTC] USER=www-data EUID=0 PID=1404015 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-user-sau-main-dev.sh
[2026-01-19 13:49:25 UTC] USER=www-data EUID=0 PID=1404033 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|user-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-user-sau-main-dev.sh
[2026-01-19 13:49:25 UTC] USER=www-data EUID=0 PID=1404042 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-user-sau-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-19 13:49:25 UTC] USER=www-data EUID=0 PID=1404060 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-user-sau-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-01-19 13:49:25 UTC] USER=www-data EUID=0 PID=1404078 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|user-sau-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-19 13:49:25 UTC] USER=www-data EUID=0 PID=1404087 ACTION=fsop ARGS=sed -i s|__ENV_ID__|user-sau-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-19 13:49:25 UTC] USER=www-data EUID=0 PID=1404096 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/data/postgresql/17/user-sau-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-19 13:49:25 UTC] USER=www-data EUID=0 PID=1404105 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-19 13:49:25 UTC] USER=www-data EUID=0 PID=1404114 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-19 13:49:25.635 P00   INFO: start command begin 2.56.0: --exec-id=1404136-6915c6a6 --log-level-console=info --log-level-file=debug --stanza=user-sau-main-dev-coordinator
2026-01-19 13:49:25.636 P00   WARN: stop file does not exist for stanza user-sau-main-dev-coordinator
2026-01-19 13:49:25.636 P00   INFO: start command end: completed successfully (5ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-19 13:49:25.680 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=1404147-8761c861 --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/user-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-user-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/user-sau-main-dev --stanza=user-sau-main-dev-coordinator
2026-01-19 13:49:25.681 P00   INFO: stanza-upgrade for stanza 'user-sau-main-dev-coordinator' on repo1
2026-01-19 13:49:25.682 P00   INFO: stanza 'user-sau-main-dev-coordinator' on repo1 is already up to date
2026-01-19 13:49:25.682 P00   INFO: stanza-upgrade command end: completed successfully (6ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-19 13:49:25 UTC] USER=www-data EUID=0 PID=1404151 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260119-134925.log
[2026-01-19 13:49:25 UTC] USER=www-data EUID=0 PID=1404160 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260119-134925.log
[2026-01-19 13:49:25 UTC] USER=www-data EUID=0 PID=1404169 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260119-134925.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-01-19 13:49:32 UTC] USER=www-data EUID=0 PID=1404279 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-1403418.log /var/log/pgbackrest/initial-backup-20260119-134925.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260119-134925.log
   2026-01-19 13:49:32.877 P00   INFO: repo1: remove expired backup 20260118-234628F
   2026-01-19 13:49:32.909 P00   INFO: repo1: remove archive path /var/lib/pgbackrest/backup/user-sau-main-dev/archive/user-sau-main-dev-coordinator/17-23
   2026-01-19 13:49:32.948 P00   INFO: repo1: 17-24 no archive to remove
   2026-01-19 13:49:32.949 P00   INFO: repo1: 17-25 no archive to remove
   2026-01-19 13:49:32.949 P00   INFO: expire command end: completed successfully (84ms)

[INFO] Current backups:
stanza: user-sau-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000003/00000001000000000000000A

        full backup: 20260119-133359F
            timestamp start/stop: 2026-01-19 13:33:59+00 / 2026-01-19 13:34:07+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.6MB, backup size: 5.6MB

        full backup: 20260119-133418F
            timestamp start/stop: 2026-01-19 13:34:18+00 / 2026-01-19 13:34:20+00
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.6MB, backup size: 5.6MB

    db (current)
        wal archive min/max (17): 000000010000000000000003/000000010000000000000006

        full backup: 20260119-134906F
            timestamp start/stop: 2026-01-19 13:49:06+00 / 2026-01-19 13:49:14+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.6MB, backup size: 5.6MB

        full backup: 20260119-134925F
            timestamp start/stop: 2026-01-19 13:49:25+00 / 2026-01-19 13:49:32+00
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.6MB, backup size: 5.6MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         user-sau-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/user-sau-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/user-sau-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=user-sau-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ Backup setup completed for coordinator and all workers

[INFO] Skipping 06-distribute-tables-canary.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 07-distribute-tables.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:49:34 UTC] USER=unknown EUID=33 PID=1404432 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-19 13:49:34 UTC] USER=unknown EUID=33 PID=1404439 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-19 13:49:34 UTC] USER=unknown EUID=33 PID=1404446 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-19 13:49:34 UTC] USER=unknown EUID=33 PID=1404453 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS TABLE DISTRIBUTION
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔐 Secure connection established
[INFO]    Host: db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432
[INFO]    Database: fastorder_user_sau_main_dev_db
[INFO]    SSL: verify-full (TLS 1.2+)
[INFO]    Timeouts: statement=120s, idle_tx=300s

[INFO] 🔍 Running preflight checks...
[INFO] Testing database connectivity...
[OK]   ✅ Database connection successful
[OK]   ✅ Connected to correct database: fastorder_user_sau_main_dev_db
[INFO] Checking Citus extension in database fastorder_user_sau_main_dev_db...
[OK]   Citus version: 13.2-1
[INFO] Checking worker registration...
[OK]   Registered workers: 1
[INFO] Worker nodes:
[INFO]                           nodename                         | nodeport | isactive | noderole 
[INFO]   ---------------------------------------------------------+----------+----------+----------
[INFO]    db-user-sau-main-dev-postgresql-worker-01.fastorder.com |     5432 | t        | primary
[INFO]   (1 row)
[INFO]   

[INFO] 📊 Starting table distribution...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Distributing: auth.login_account
[INFO] Description: User authentication table - distributed by region for tenant isolation
[INFO] Shard key: region_hint
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ⏭️  Table does not exist, skipping

[INFO] ═══════════════════════════════════════════════════════════════════════════════
[OK]   ✅ All tables distributed successfully!
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 📊 Citus Cluster Summary:

[INFO] Distributed tables:
[INFO]              table           |    type     | shard_key | shards |  size   
[INFO]   ---------------------------+-------------+-----------+--------+---------
[INFO]    "user".contract_key       | reference   | <none>    |      1 | 16 kB
[INFO]    "user".contract_type      | reference   | <none>    |      1 | 16 kB
[INFO]    "user".contract_term_json | distributed | id        |     32 | 512 kB
[INFO]    "user".contract_term_vars | distributed | id        |     32 | 1792 kB
[INFO]   (4 rows)
[INFO]   

[INFO] Worker capacity:
[INFO]    worker | total_shards | total_size 
[INFO]   --------+--------------+------------
[INFO]   (0 rows)
[INFO]   

[OK]   Citus table distribution complete

[INFO] Skipping 08-distribute-tables-rollback.sh (rollback script - run manually only)
[INFO] Skipping 09-distribute-tables-test.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 10-setup-cdc.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] CDC PIPELINE SETUP (Debezium + Elasticsearch Sink)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Log file: /var/log/fastorder/cdc/10-setup-cdc-*.log

[INFO] Running CDC setup for identifier: coordinator
[2026-01-19 13:49:40] ==========================================
[2026-01-19 13:49:40] CDC SETUP SCRIPT STARTED
[2026-01-19 13:49:40] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260119_134940.log
[2026-01-19 13:49:40] ==========================================
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-19 13:49:40] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:49:40]   CDC Pipeline Setup (Debezium + ES Sink)
[2026-01-19 13:49:40] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:49:40]   Environment: user-sau-main-dev
[2026-01-19 13:49:40]   Identifier:  coordinator
[2026-01-19 13:49:40]   Service:     user
[2026-01-19 13:49:40] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:49:40] 📂 CDC_BASE_DIR exists: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc
[2026-01-19 13:49:40] Looking for service folder: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user
[2026-01-19 13:49:40] 
[2026-01-19 13:49:40] 📂 Found CDC configuration for service: user
[2026-01-19 13:49:40] Scanning for subservice directories in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user
[2026-01-19 13:49:40] Found subservice: contracts, checking for steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps
[2026-01-19 13:49:40] 
[2026-01-19 13:49:40] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:49:40]   Setting up CDC for: user/contracts
[2026-01-19 13:49:40] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:49:40] Found 8 step script(s) in /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps
[2026-01-19 13:49:40] 
[2026-01-19 13:49:40] 🔧 Running: 00-create-eav-tables.sh
[2026-01-19 13:49:40]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/00-create-eav-tables.sh
[2026-01-19 13:49:40]    Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Create EAV Tables for CDC Pipeline
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Identifier:  coordinator
  Tables:      user.contracts_int, user.contracts_json
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📝 Step 1: Creating EAV tables...
📥 Executing SQL...
  BEGIN
  CREATE TABLE
  CREATE INDEX
  CREATE INDEX
  CREATE INDEX
  CREATE FUNCTION
  psql:/tmp/create-eav-tables-user-sau-main-dev.sql:61: NOTICE:  trigger "trg_contracts_int_updated_at" for relation "user.contracts_int" does not exist, skipping
  DROP TRIGGER
  CREATE TRIGGER
  ALTER TABLE
  CREATE TABLE
  CREATE INDEX
  CREATE INDEX
  CREATE INDEX
  CREATE INDEX
  CREATE FUNCTION
  psql:/tmp/create-eav-tables-user-sau-main-dev.sql:120: NOTICE:  trigger "trg_contracts_json_updated_at" for relation "user.contracts_json" does not exist, skipping
  DROP TRIGGER
  CREATE TRIGGER
  ALTER TABLE
  COMMIT
  psql:/tmp/create-eav-tables-user-sau-main-dev.sql:161: NOTICE:  Created publication with all tables
  DO
  pubname         | schemaname |   tablename
  ------------------------+------------+----------------
  cdc_pub_user_contracts | user       | contracts
  cdc_pub_user_contracts | user       | contracts_int
  cdc_pub_user_contracts | user       | contracts_json
  (3 rows)
  
✅ EAV tables created

🔍 Step 2: Verifying tables...

📊 Table: user.contracts_int
                                Table "user.contracts_int"
   Column    |           Type           | Collation | Nullable |         Default          
-------------+--------------------------+-----------+----------+--------------------------
 id          | uuid                     |           | not null | utils.uuid_generate_v7()
 tenant_id   | character varying(100)   |           | not null | 
 contract_id | character(36)            |           | not null | 
 key         | character varying(100)   |           | not null | 
 value_int   | integer                  |           | not null | 
 created_at  | timestamp with time zone |           | not null | now()
 updated_at  | timestamp with time zone |           | not null | now()
Indexes:
    "contracts_int_pkey" PRIMARY KEY, btree (id)
    "idx_contracts_int_contract_id" btree (contract_id)
    "idx_contracts_int_key" btree (key)
    "idx_contracts_int_tenant_contract" btree (tenant_id, contract_id)
    "uq_contracts_int_contract_key" UNIQUE CONSTRAINT, btree (contract_id, key)
Foreign-key constraints:
    "fk_contracts_int_contract" FOREIGN KEY (contract_id) REFERENCES "user".contracts(id) ON DELETE CASCADE
Publications:
    "cdc_pub_user_contracts"

📊 Table: user.contracts_json
                               Table "user.contracts_json"
   Column    |           Type           | Collation | Nullable |         Default          
-------------+--------------------------+-----------+----------+--------------------------
 id          | uuid                     |           | not null | utils.uuid_generate_v7()
 tenant_id   | character varying(100)   |           | not null | 
 contract_id | character(36)            |           | not null | 
 key         | character varying(100)   |           | not null | 
 value_json  | jsonb                    |           | not null | 
 created_at  | timestamp with time zone |           | not null | now()
 updated_at  | timestamp with time zone |           | not null | now()
Indexes:
    "contracts_json_pkey" PRIMARY KEY, btree (id)
    "idx_contracts_json_contract_id" btree (contract_id)
    "idx_contracts_json_key" btree (key)
    "idx_contracts_json_tenant_contract" btree (tenant_id, contract_id)
    "idx_contracts_json_value_gin" gin (value_json)
    "uq_contracts_json_contract_key" UNIQUE CONSTRAINT, btree (contract_id, key)
Foreign-key constraints:
    "fk_contracts_json_contract" FOREIGN KEY (contract_id) REFERENCES "user".contracts(id) ON DELETE CASCADE
Publications:

📊 Publication Tables:
user.contracts
user.contracts_int
user.contracts_json

📝 Step 3: Sample data commands (for testing)...

-- Insert sample INT attributes (tenant_id must match parent contract)
INSERT INTO "user".contracts_int (tenant_id, contract_id, "key", value_int)
VALUES
    ('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'max_users', 100),
    ('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'credit_limit', 50000),
    ('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'tier_level', 2)
ON CONFLICT (contract_id, "key")
DO UPDATE SET value_int = EXCLUDED.value_int, updated_at = NOW();

-- Insert sample JSON attributes (tenant_id must match parent contract)
INSERT INTO "user".contracts_json (tenant_id, contract_id, "key", value_json)
VALUES
    ('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'metadata', '{"lang":"en","tier":"gold"}'::jsonb),
    ('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'settings', '{"notifications":true,"theme":"dark"}'::jsonb),
    ('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'permissions', '{"admin":true,"export":true}'::jsonb)
ON CONFLICT (contract_id, "key")
DO UPDATE SET value_json = EXCLUDED.value_json, updated_at = NOW();


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  EAV Tables Created Successfully
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Tables:
    - user.contracts_int
    - user.contracts_json

  Publication: cdc_pub_user_contracts

  Next Steps:
    1. Update Debezium connector table.include.list
    2. Setup ksqlDB pipeline (05-setup-ksqldb-pipeline.sh)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:49:41] ✅ Completed: 00-create-eav-tables.sh
[2026-01-19 13:49:41] 
[2026-01-19 13:49:41] 🔧 Running: 00b-migrate-tenant-id.sh
[2026-01-19 13:49:41]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/00b-migrate-tenant-id.sh
[2026-01-19 13:49:41]    Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Migration: Add tenant_id to EAV Tables
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Identifier:  coordinator
  Tables:      user.contracts_int, user.contracts_json
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔍 Pre-flight: Checking current state...
✅ tenant_id column already exists in both tables
✅ tenant_id is already NOT NULL - migration complete
[2026-01-19 13:49:42] ✅ Completed: 00b-migrate-tenant-id.sh
[2026-01-19 13:49:42] 
[2026-01-19 13:49:42] 🔧 Running: 01-setup-debezium-user-contracts.sh
[2026-01-19 13:49:42]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/01-setup-debezium-user-contracts.sh
[2026-01-19 13:49:42]    Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Debezium CDC Setup (User Contracts)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Identifier:  coordinator
  Table:       user.contracts
  Privacy:     Minimal user index (GDPR compliant)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔍 Verifying Kafka infrastructure...
✅ db-user-sau-main-dev-postgresql.fastorder.com resolves to 10.100.1.231
🔐 psql will use client cert for mTLS.
🔐 Retrieving credentials from secrets vault...
✅ Credentials retrieved from secrets vault
🔐 Writing Debezium credentials to FileConfigProvider secrets file...
[2026-01-19 13:49:46 UTC] USER=www-data EUID=0 PID=1405319 ACTION=passthru ARGS=sed -i s|^debezium.database.password=.*|debezium.database.password=AlBKuoQv6SJDHYCwpHFMFmqWS| /opt/kafka/secrets/user-sau-main-dev/coordinator/connector-secrets.properties
✅ Updated Debezium credentials in /opt/kafka/secrets/user-sau-main-dev/coordinator/connector-secrets.properties
🔐 Syncing debezium_user password in PostgreSQL...
✅ debezium_user password synchronized
🔍 Checking PostgreSQL SSL status...
✅ Server SSL is ON.
🔧 Applying schema, publication & grants over TLS…
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

DROP PUBLICATION
CREATE PUBLICATION
NOTICE:  Added user.contracts_int to publication
NOTICE:  Added user.contracts_json to publication
DO
GRANT
GRANT
GRANT
GRANT
GRANT
✅ Publication & grants done.
⏳ Waiting for Kafka Connect @ https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083/connectors…
[2026-01-19 13:49:46] 🔗 Waiting for Kafka Connect at: https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083
[2026-01-19 13:49:46] ⏳ Waiting for HTTP endpoint: https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083
[2026-01-19 13:49:46]    Expected codes: 200,500, timeout: 300s
[2026-01-19 13:49:46] ✅ HTTP endpoint ready: https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083 (code: 200, took: 0s)
[2026-01-19 13:49:46] 🔄 Testing Connect worker readiness...
[2026-01-19 13:49:46] ✅ Kafka Connect worker ready
🧹 Cleaning up existing Debezium connector and slot (if any)...
   Step 0a: Also resetting ES Sink connector offsets (required for coordinated reset)...
   → Deleting ES Sink connector offsets...
   → Creating temporary ES Sink placeholder for offset deletion...
{"error_code":400,"message":"Connector configuration is invalid and contains the following 2 error(s):\nCould not connect to Elasticsearch. Error message: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused\nFailed to create client to verify connection. java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused\nYou can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`"}{"error_code":404,"message":"Unknown connector pg_user_sau_main_dev_coordinator_user_contracts_es_sink"}   ⚠️  ES Sink offset deletion returned HTTP 404 (may be OK if no offsets existed)
   → Deleting ES Sink connector...
{"error_code":404,"message":"Connector pg_user_sau_main_dev_coordinator_user_contracts_es_sink not found"}   ✓ ES Sink connector cleanup complete
   Step 0b: Clearing stale Debezium connector offsets from Kafka Connect...
   → Stopping connector pg_user_sau_main_dev_user_contracts_debezium...
   → Deleting connector offsets (forces fresh snapshot)...
   ✓ Connector offsets deleted successfully (HTTP 200)
   Step 1: Deleting Debezium connector...
   Deleting connector: pg_user_sau_main_dev_user_contracts_debezium (attempt 1/10)
   ✓ Connector pg_user_sau_main_dev_user_contracts_debezium confirmed deleted
   Step 2: Waiting for replication slot to become inactive...
   ✓ Slot slot_user_sau_main_dev_user_contracts does not exist (clean state)
   Step 3: Dropping replication slot...
   ✓ Slot slot_user_sau_main_dev_user_contracts already dropped
   Step 4: Final verification...
✅ Cleanup complete - environment is clean for fresh CDC snapshot
🔐 Checking Debezium SSL certificate permissions...
🔍 Validating Debezium SSL certificates...
🔐 Connector will use mTLS to Postgres.
  ✓ Certificate: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt
  ✓ Key: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der
  ✓ Root CA: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
📤 Upserting connector: PUT https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_user_sau_main_dev_user_contracts_debezium/config
   Attempt 1/5: Sending PUT request to Kafka Connect...
   (This may take up to 60s as Connect validates the configuration)
   ✅ Success (HTTP 201)

🌐 HTTP Response: 201
✅ Connector upserted.
🔄 Verifying connector task startup...
✅ Debezium connector task is RUNNING
ℹ️  Source table user.contracts has 0 rows.
ℹ️  Snapshot will be metadata-only; offsets may stay empty until first change.
⏳ Waiting for Debezium initial snapshot to complete...
   📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
   📊 Debezium snapshot status: unknown
   ⏳ Snapshot in progress... (0s elapsed)
   ⏳ Snapshot in progress... (5s elapsed)
   ⏳ Snapshot in progress... (10s elapsed)
   📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
   📊 Debezium snapshot status: unknown
   ⏳ Snapshot in progress... (15s elapsed)
   ⏳ Snapshot in progress... (20s elapsed)
   ⏳ Snapshot in progress... (25s elapsed)
   📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
   📊 Debezium snapshot status: unknown
   ⏳ Snapshot in progress... (30s elapsed)
   ⏳ Snapshot in progress... (35s elapsed)
   ⏳ Snapshot in progress... (40s elapsed)
   📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
   📊 Debezium snapshot status: unknown
   ⏳ Snapshot in progress... (45s elapsed)
   ⏳ Snapshot in progress... (50s elapsed)
   ⏳ Snapshot in progress... (55s elapsed)
   📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
   📊 Debezium snapshot status: unknown
   ⏳ Snapshot in progress... (60s elapsed)
   ⏳ Snapshot in progress... (65s elapsed)
   ⏳ Snapshot in progress... (70s elapsed)
   📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
   📊 Debezium snapshot status: unknown
   ⏳ Snapshot in progress... (75s elapsed)
   ⏳ Snapshot in progress... (80s elapsed)
   ⏳ Snapshot in progress... (85s elapsed)
   📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
   📊 Debezium snapshot status: unknown
   ⏳ Snapshot in progress... (90s elapsed)
   ⏳ Snapshot in progress... (95s elapsed)
   ⏳ Snapshot in progress... (100s elapsed)
   📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
   📊 Debezium snapshot status: unknown
   ⏳ Snapshot in progress... (105s elapsed)
   ⏳ Snapshot in progress... (110s elapsed)
   ⏳ Snapshot in progress... (115s elapsed)

⚠️  Snapshot wait timeout (120s) on EMPTY table.
   Offsets are still empty, but source table has 0 rows.
   Proceeding anyway – CDC health will be verified by test inserts.
✅ Debezium connector is RUNNING after snapshot
🔍 Final verification: Checking Debezium offsets are recorded...
   ℹ️  Source table has 0 rows - skipping offset verification
✅ Debezium connector verified RUNNING (empty source table)
🔄 Phase 2: Updating connector to snapshot.mode=initial...
✅ Connector updated to snapshot.mode=initial (HTTP 200)
✅ Connector verified RUNNING after Phase 2 update
✅ Debezium connector configured successfully (two-phase snapshot complete)

==================================================================
MULTI-TABLE CDC Pipeline Configuration
==================================================================
   Tables:
     - user.contracts (main table)
     - user.contracts_int (EAV integer attributes)
     - user.contracts_json (EAV JSON attributes)

   Topics:
     - cdc.user.contracts
     - cdc.user.contracts_int
     - cdc.user.contracts_json

   COLUMN EXCLUSION (raw PII never leaves PostgreSQL):
     user.contracts.email,user.contracts.phone

   CAPTURED (safe for Kafka/ES):
     id (PK), tenant_id, home_region, username,
     display_name, email_hash, phone_hash, country_code,
     region_code, tags, segments, contract info

   DATA FLOW (Multi-Table CDC with ksqlDB Join):
     PostgreSQL Tables (1:N)
         ↓ Debezium (CDC per table)
         ↓ Kafka Topics (3 separate topics)
         ↓ ksqlDB (pivot + join → flat document)
         ↓ Compacted Topic (search.user.contracts.v1)
         ↓ ES Sink (UPSERT)
         ↓ Elasticsearch (flat search index)

   NEXT STEPS:
     1. Run 00-create-eav-tables.sh (if not done)
     2. Run 05-setup-ksqldb-pipeline.sh
     3. Run 06-setup-es-sink-ksqldb.sh
     4. Run 07-test-multi-table-cdc.sh
==================================================================
[2026-01-19 13:52:12] ✅ Completed: 01-setup-debezium-user-contracts.sh
[2026-01-19 13:52:12] 
[2026-01-19 13:52:12] 🔧 Running: 01b-install-ksqldb.sh
[2026-01-19 13:52:12]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/01b-install-ksqldb.sh
[2026-01-19 13:52:12]    Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ksqlDB Installation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Identifier:  coordinator
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  VM_IP:     10.100.1.234
  FQDN:      eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com

📦 Step 1: Checking Confluent Platform installation...
✅ ksqlDB already installed (version: )

📁 Step 2: Creating directories...
[2026-01-19 13:52:15 UTC] USER=www-data EUID=0 PID=1409537 ACTION=fsop ARGS=mkdir -p /var/lib/ksqldb/user-sau-main-dev/coordinator
[2026-01-19 13:52:15 UTC] USER=www-data EUID=0 PID=1409580 ACTION=fsop ARGS=mkdir -p /etc/ksqldb/user-sau-main-dev/coordinator
[2026-01-19 13:52:15 UTC] USER=www-data EUID=0 PID=1409603 ACTION=fsop ARGS=chown -R kafka:kafka /var/lib/ksqldb/user-sau-main-dev/coordinator /var/log/ksqldb/user-sau-main-dev/coordinator /etc/ksqldb/user-sau-main-dev/coordinator
✅ Directories created

⚙️  Step 3: Generating ksqlDB configuration...
[2026-01-19 13:52:15 UTC] USER=www-data EUID=0 PID=1409642 ACTION=fsop ARGS=mv /tmp/ksql-server-user-sau-main-dev.properties /etc/ksqldb/user-sau-main-dev/coordinator/ksql-server.properties
[2026-01-19 13:52:16 UTC] USER=www-data EUID=0 PID=1409684 ACTION=fsop ARGS=chmod 640 /etc/ksqldb/user-sau-main-dev/coordinator/ksql-server.properties
✅ Configuration generated: /etc/ksqldb/user-sau-main-dev/coordinator/ksql-server.properties

🔧 Step 4: Creating systemd service...
[2026-01-19 13:52:16 UTC] USER=www-data EUID=0 PID=1409729 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-19 13:52:16 UTC] USER=www-data EUID=0 PID=1409790 ACTION=passthru ARGS=systemctl enable ksqldb-user-sau-main-dev-coordinator.service
✅ Systemd service created: ksqldb-user-sau-main-dev-coordinator.service

🚀 Step 5: Starting ksqlDB service...
🔍 Checking Kafka broker connectivity...
✅ Kafka broker is accessible
[2026-01-19 13:52:17 UTC] USER=www-data EUID=0 PID=1409851 ACTION=passthru ARGS=systemctl start ksqldb-user-sau-main-dev-coordinator.service
✅ ksqlDB service started
⏳ Waiting for ksqlDB to be ready...
✅ ksqlDB is ready!


🔍 Step 6: Verifying installation...

📊 Service Status:
[2026-01-19 13:52:17 UTC] USER=www-data EUID=0 PID=1409875 ACTION=passthru ARGS=systemctl status ksqldb-user-sau-main-dev-coordinator.service --no-pager -l
● ksqldb-user-sau-main-dev-coordinator.service - ksqlDB Server (user-sau-main-dev coordinator)
     Loaded: loaded (/etc/systemd/system/ksqldb-user-sau-main-dev-coordinator.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2026-01-19 13:19:34 UTC; 32min ago
       Docs: https://docs.ksqldb.io/
   Main PID: 1296228 (java)
      Tasks: 112 (limit: 19051)
     Memory: 527.4M
        CPU: 2min 23.297s
     CGroup: /system.slice/ksqldb-user-sau-main-dev-coordinator.service
             └─1296228 java -cp "/usr/share/java/ksqldb/*:/usr/share/java/rest-utils/*:/usr/share/java/confluent-common/*:" -Xms256m -Xmx512m -server -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:NewRatio=1 -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dksql.log.dir=/var/log/ksqldb/user-sau-main-dev/coordinator -Dlog4j.configuration=file:/etc/ksqldb/log4j.properties -Dksql.server.install.dir=/usr "-Xlog:gc*:file=/var/log/ksqldb/user-sau-main-dev/coordinator/ksql-server-gc.log:time,tags:filecount=10,filesize=102400" io.confluent.ksql.rest.server.KsqlServerMain /etc/ksqldb/user-sau-main-dev/coordinator/ksql-server.properties

Jan 19 13:51:58 web-03 ksql-server-start[1296228]: [2026-01-19 13:51:58,630] INFO stream-thread [_confluent-ksql-user-sau-main-dev_ksqldb_coordinatorquery_CTAS_CONTRACTS_TABLE_291-7e89eccb-ceda-4a23-a406-dd5a4bbe0bc9-StreamThread-1] Processed 0 total records, ran 0 punctuators, and committed 0 total tasks since the last update (org.apache.kafka.streams.processor.internals.StreamThread:882)
Jan 19 13:51:58 web-03 ksql-server-start[1296228]: [2026-01-19 13:51:58,630] INFO stream-thread [_confluent-ksql-user-sau-main-dev_ksqldb_coordinatorquery_CTAS_CONTRACTS_JSON_AGG_289-f4d5c8dc-cd18-4200-9918-d2dee68acf57-StreamThread-3] Processed 0 total records, ran 0 punctuators, and committed 0 total tasks since the last update (org.apache.kafka.streams.processor.internals.StreamThread:882)
Jan 19 13:51:58 web-03 ksql-server-start[1296228]: [2026-01-19 13:51:58,649] INFO stream-thread [_confluent-ksql-user-sau-main-dev_ksqldb_coordinatorquery_CTAS_CONTRACTS_JSON_AGG_289-f4d5c8dc-cd18-4200-9918-d2dee68acf57-StreamThread-1] Processed 0 total records, ran 0 punctuators, and committed 0 total tasks since the last update (org.apache.kafka.streams.processor.internals.StreamThread:882)

📊 ksqlDB Info:
{
  "KsqlServerInfo": {
    "version": "7.6.5",
    "kafkaClusterId": "[2026-01-15 17:36:55 UTC] USER=www-data EUID=0 PID=455661 ACTION=passthru ARGS=bash -c cat /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/kafka_kraft_cluster_id\nvGsJvzNtQGKG1HQPRIaTPQ",
    "ksqlServiceId": "user-sau-main-dev_ksqldb_coordinator",
    "serverStatus": "RUNNING"
  }
}
✅ ksqlDB is responding

📡 Step 7: Registering ksqlDB to Observability API...
🔄 Registering ksqlDB node to observability dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       ksqlDB
[INFO]   Identifier:        user-sau-main-dev-ksqldb-coordinator
[INFO]   Identifier Parent: eventbus
[INFO]   IP:                10.100.1.234
[INFO]   Port:              8088
[INFO]   FQDN:              eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 07aaaced-f263-402d-90c8-50c9a9c0ff5c
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
✅ ksqlDB registered successfully

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ksqlDB Installation Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Service:  ksqldb-user-sau-main-dev-coordinator
  VM_IP:    10.100.1.234
  FQDN:     eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com
  Port:     8088
  Config:   /etc/ksqldb/user-sau-main-dev/coordinator/ksql-server.properties
  Data:     /var/lib/ksqldb/user-sau-main-dev/coordinator
  Logs:     /var/log/ksqldb/user-sau-main-dev/coordinator

  Dashboard:
    https://skeleton.dev.fastorder.com/dashboard/monitoring/environment2/<env-id>/service/ksqldb

  CLI Access (with SSL):
    ksql --ssl https://eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com:8088

  REST API (HTTPS):
    curl -k https://eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com:8088/info
    curl -k https://eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com:8088/ksql -H 'Content-Type: application/vnd.ksql.v1+json' -d '{"ksql": "SHOW STREAMS;"}'
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:52:18] ✅ Completed: 01b-install-ksqldb.sh
[2026-01-19 13:52:18] 
[2026-01-19 13:52:18] 🔧 Running: 02-setup-ksqldb-pipeline.sh
[2026-01-19 13:52:18]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/02-setup-ksqldb-pipeline.sh
[2026-01-19 13:52:18]    Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ksqlDB CDC Pipeline Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Tables:      user.contracts, contracts_int, contracts_json
  Output:      user_sau_main_dev_user_contracts
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔍 Step 0: Checking ksqlDB availability...
✅ ksqlDB is running (version: 7.6.5)

📦 Step 1: Creating compacted output topic...
📥 Creating compacted topic: user_sau_main_dev_user_contracts
[2026-01-19 13:52:21 UTC] USER=www-data EUID=0 PID=1410432 ACTION=passthru ARGS=sudo -u kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com:9092 --command-config /tmp/kafka-client-user-sau-main-dev.properties --create --topic user_sau_main_dev_user_contracts --partitions 12 --replication-factor 1 --config cleanup.policy=compact --config min.compaction.lag.ms=0 --config delete.retention.ms=86400000 --config segment.ms=3600000
Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:561)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:512)
	at org.apache.kafka.clients.admin.Admin.create(Admin.java:137)
	at org.apache.kafka.tools.TopicCommand$TopicService.createAdminClient(TopicCommand.java:456)
	at org.apache.kafka.tools.TopicCommand$TopicService.<init>(TopicCommand.java:445)
	at org.apache.kafka.tools.TopicCommand.execute(TopicCommand.java:101)
	at org.apache.kafka.tools.TopicCommand.mainNoExit(TopicCommand.java:90)
	at org.apache.kafka.tools.TopicCommand.main(TopicCommand.java:85)
Caused by: org.apache.kafka.common.KafkaException: Failed to create new NetworkClient
	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:255)
	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:190)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:545)
	... 7 more
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /opt/kafka/secrets/user-sau-main-dev/coordinator/kafka.client.keystore.p12 of type JKS
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:380)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:352)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:302)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:162)
	at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:147)
	at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:100)
	at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:70)
	at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:193)
	at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:82)
	at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:120)
	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:224)
	... 9 more
Caused by: java.io.IOException: keystore password was incorrect
	at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159)
	at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:242)
	at java.base/java.security.KeyStore.load(KeyStore.java:1473)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:377)
	... 19 more
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
	... 23 more

✅ Topic created with compaction enabled

🧹 Step 1b: Checking for existing ksqlDB objects to clean up...
✅ No existing objects found - proceeding with fresh creation

📝 Step 2: Generating ksqlDB DDL...
✅ DDL generated: /tmp/ksql-user-contracts-user-sau-main-dev.ksql

🚀 Step 3: Executing ksqlDB DDL...
📋 Executing DDL statements...
  → CREATE STREAM IF NOT EXISTS contracts_stream (...
    ⚠️  
  → CREATE STREAM IF NOT EXISTS contracts_int_stream (...
    ⚠️  
  → CREATE STREAM IF NOT EXISTS contracts_json_stream (...
    ⚠️  
  → CREATE TABLE IF NOT EXISTS contracts_int_agg...
    ⚠️  
  → SELECT...
    ⚠️  
  → CREATE TABLE IF NOT EXISTS contracts_json_agg...
    ⚠️  
  → SELECT...
    ⚠️  
  → CREATE TABLE IF NOT EXISTS contracts_tbl...
    ⚠️  
  → SELECT...
    ⚠️  
  → CREATE TABLE IF NOT EXISTS user_search_doc_raw...
    ⚠️  
  → SELECT...
    ⚠️  
  → CREATE STREAM IF NOT EXISTS user_search_doc_keyed...
    ⚠️  
  → SELECT...
    ⚠️  
  → PARTITION BY `doc_id`...
    ⚠️  

🔍 Step 4: Verifying ksqlDB objects...

📊 Streams:

📊 Tables:

📊 Running Queries:
  Active queries: 

💾 Step 5: Saving DDL for reference...
[2026-01-19 13:52:25 UTC] USER=www-data EUID=0 PID=1411215 ACTION=passthru ARGS=mkdir -p /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/ksqldb
[2026-01-19 13:52:25 UTC] USER=www-data EUID=0 PID=1411236 ACTION=passthru ARGS=cp /tmp/ksql-user-contracts-user-sau-main-dev.ksql /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/ksqldb/user-contracts-pipeline.ksql
✅ DDL saved to: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/ksqldb/user-contracts-pipeline.ksql

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ksqlDB Pipeline Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Input Topics:
    - user_sau_main_dev_cdc.user.contracts
    - user_sau_main_dev_cdc.user.contracts_int
    - user_sau_main_dev_cdc.user.contracts_json

  Output Topic:
    - user_sau_main_dev_user_contracts (compacted)

  ksqlDB Objects:
    - Streams: contracts_stream, contracts_int_stream, contracts_json_stream
    - Tables: contracts_tbl (keyed by doc_id)
    - Tables: contracts_int_agg, contracts_json_agg (keyed by doc_id)
    - Tables: user_search_doc_raw (joined table)
    - Streams: user_search_doc_keyed (final output)

  Join Key: doc_id = CONCAT(tenant_id, ':', contract_id)

  Next Steps:
    1. Update ES Sink to consume from: user_sau_main_dev_user_contracts
    2. Test with data insertion
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:52:25] ✅ Completed: 02-setup-ksqldb-pipeline.sh
[2026-01-19 13:52:25] 
[2026-01-19 13:52:25] 🔧 Running: 03-setup-es-sink-ksqldb.sh
[2026-01-19 13:52:25]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/03-setup-es-sink-ksqldb.sh
[2026-01-19 13:52:25]    Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ES Sink Connector (ksqlDB Joined Topic)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Identifier:  coordinator
  Input:       user_sau_main_dev_user_contracts
  Output:      user_sau_main_dev_user_contracts (index)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔐 Step 1: Getting Elasticsearch credentials...
✅ Elasticsearch credentials loaded
✅ SSL passwords loaded

🔒 Step 1c: Ensuring ES client keystore and truststore are properly configured...
📦 Creating ES client keystore from PEM certificates...
[2026-01-19 13:52:29 UTC] USER=www-data EUID=0 PID=1411448 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/user-sau-main-dev/coordinator/es-client.keystore.p12
[2026-01-19 13:52:29 UTC] USER=www-data EUID=0 PID=1411470 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/es-client.keystore.p12
✅ ES client keystore created: /opt/kafka/secrets/user-sau-main-dev/coordinator/es-client.keystore.p12
🔑 Checking if ES HTTP CA is in truststore...
📦 Adding ES HTTP CA to truststore...
[2026-01-19 13:52:30 UTC] USER=www-data EUID=0 PID=1411527 ACTION=keytool ARGS=-importcert -alias es-http-ca -file /etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt -keystore /opt/kafka/secrets/user-sau-main-dev/coordinator/truststore.jks -storepass dV4AfOMsnuZ0cdEeyvgt1IHch08Rnm0j -noprompt
Certificate was added to keystore
✅ ES HTTP CA added to truststore

🔐 Step 1b: Writing credentials to FileConfigProvider secrets file...
[2026-01-19 13:52:31 UTC] USER=www-data EUID=0 PID=1411574 ACTION=passthru ARGS=sed -i s|^elasticsearch.connection.username=.*|elasticsearch.connection.username=elastic| /opt/kafka/secrets/user-sau-main-dev/coordinator/connector-secrets.properties
[2026-01-19 13:52:31 UTC] USER=www-data EUID=0 PID=1411601 ACTION=passthru ARGS=sed -i s|^elasticsearch.connection.password=.*|elasticsearch.connection.password=silIukM1=kC+UVuB0SHB| /opt/kafka/secrets/user-sau-main-dev/coordinator/connector-secrets.properties
✅ Updated Elasticsearch credentials in /opt/kafka/secrets/user-sau-main-dev/coordinator/connector-secrets.properties

🔄 Step 2: Checking for existing connector...
⚠️  Existing connector found, deleting...
✅ Existing connector deleted

📝 Step 3: Creating ES Sink connector configuration...

🚀 Step 4: Deploying ES Sink connector...
✅ Connector deployed successfully
⏳ Waiting for connector to start...

🔍 Step 5: Verifying connector status...

✅ Connector State: RUNNING
✅ Task State: RUNNING

🔒 Step 6: Creating Elasticsearch ingest pipeline (remove sensitive fields)...
✅ Ingest pipeline created: user_sau_main_dev_user_contracts-pipeline
   Removes: userid, user_id, email_hash, phone_hash, password_hash, etc.

📋 Step 7a: Creating ILM Policy...
✅ ILM policy created: user_sau_main_dev_user_contracts_ilm_policy
   Rollover: 30d or 50GB | Warm: 30d | Delete: 365d

📋 Step 7b: Creating Elasticsearch index template with ILM...
✅ Index template created: user_sau_main_dev_user_contracts_template

🔗 Step 7c: Creating initial ILM index with alias...
✅ Initial ILM index already exists: user_sau_main_dev_user_contracts-000001

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ES Sink Connector Setup Complete (with ILM)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Connector:  pg_user_sau_main_dev_coordinator_ksqldb_contracts_es_sink
  Input:      user_sau_main_dev_user_contracts
  Index:      user_sau_main_dev_user_contracts (alias → user_sau_main_dev_user_contracts-000001)
  ILM Policy: user_sau_main_dev_user_contracts_ilm_policy
  Pipeline:   user_sau_main_dev_user_contracts-pipeline
  DLQ:        dlq.user_sau_main_dev_user_contracts

  ILM Configuration:
    Hot:    rollover at 30d or 50GB (priority 100)
    Warm:   after 30d, shrink to 1 shard, forcemerge (priority 50)
    Delete: after 365d

  Key Configuration:
    write.method:            UPSERT
    key.ignore:              false (Kafka key → ES _id)
    behavior.on.null.values: DELETE (tombstone → delete)
    default_pipeline:        user_sau_main_dev_user_contracts-pipeline (removes userid, etc.)

  Data Minimization (GDPR Compliance):
    Pipeline removes: userid, user_id, email_hash, phone_hash,
                      password_hash, password_salt, mfa_secret

  Verify Commands:
    # Connector status
    curl -k https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_user_sau_main_dev_coordinator_ksqldb_contracts_es_sink/status | jq

    # Document count (via alias)
    curl -k -u elastic:PASS https://10.100.1.152:9200/user_sau_main_dev_user_contracts/_count

    # Check ILM status
    curl -k -u elastic:PASS https://10.100.1.152:9200/user_sau_main_dev_user_contracts-000001/_ilm/explain

    # List indices with alias
    curl -k -u elastic:PASS https://10.100.1.152:9200/_cat/aliases?v
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:52:40] ✅ Completed: 03-setup-es-sink-ksqldb.sh
[2026-01-19 13:52:40] 
[2026-01-19 13:52:40] 🔧 Running: 04-test-multi-table-cdc.sh
[2026-01-19 13:52:40]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/04-test-multi-table-cdc.sh
[2026-01-19 13:52:40]    Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Multi-Table CDC Pipeline Test (User Contracts)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Tables:      user.contracts, user.contracts_int, user.contracts_json
  Pipeline:    Debezium → ksqlDB → ES Sink → Elasticsearch
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📝 Test 1: INSERT into user.contracts
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Test UUID: 019bd687-7a00-7099-8c80-d304e44defda
🔧 Test Tenant: cdc_test_tenant_1768830761
INSERT 0 1
✅ Contract inserted

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📝 Test 2: INSERT into user.contracts_int
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
INSERT 0 3
✅ INT attributes inserted (tenant_id=cdc_test_tenant_1768830761, max_users=100, credit_limit=50000, tier_level=2)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📝 Test 3: INSERT into user.contracts_json
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
INSERT 0 3
✅ JSON attributes inserted (tenant_id=cdc_test_tenant_1768830761, metadata, settings, permissions)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔍 Test 4: Verify joined document in Elasticsearch
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📋 Expected doc_id: cdc_test_tenant_1768830761:019bd687-7a00-7099-8c80-d304e44defda
⏳ Waiting for CDC propagation (up to 60s)...

   Polling... 3s
   Polling... 6s
   Polling... 9s
   Polling... 12s
   Polling... 15s
   Polling... 18s
   Polling... 21s
   Polling... 24s
   Polling... 27s
   Polling... 30s
   Polling... 33s
   Polling... 36s
   Polling... 39s
   Polling... 42s
   Polling... 45s
   Polling... 48s
   Polling... 51s
   Polling... 54s
   Polling... 57s
   Polling... 60s
⚠️  Document not found after 60s

📊 Troubleshooting:
   - Check ksqlDB queries: SHOW QUERIES;
   - Check topic: kafka-console-consumer --topic search.user.contracts.v1
   - Check ES Sink status: curl .../user_sau_main_dev_search_contracts/_count

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📝 Test 5: UPDATE and verify ES update
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
UPDATE 1
✅ Updated max_users to 500
⏳ Waiting for update propagation (15s)...
⚠️  ES document not yet updated (current: NOT_FOUND, expected: 500)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🗑️  Test 6: DELETE and verify ES removal
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
DELETE 1
✅ Contract deleted (CASCADE will delete EAV rows)
⏳ Waiting for delete propagation (15s)...
⚠️  Document still exists (tombstone may be pending)
   Compaction and tombstone processing may take longer

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Multi-Table CDC Test Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Test UUID:    019bd687-7a00-7099-8c80-d304e44defda
  Test Tenant:  cdc_test_tenant_1768830761
  Doc ID:       cdc_test_tenant_1768830761:019bd687-7a00-7099-8c80-d304e44defda
  Index:        user_sau_main_dev_search_contracts

  Pipeline:
    user.contracts      → Debezium → Kafka
    user.contracts_int  → Debezium → Kafka  →  ksqlDB  → Compacted Topic → ES
    user.contracts_json → Debezium → Kafka  (pivot+join)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:54:13] ✅ Completed: 04-test-multi-table-cdc.sh
[2026-01-19 13:54:13] 
[2026-01-19 13:54:13] 🔧 Running: 05-verify-cdc.sh
[2026-01-19 13:54:13]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/05-verify-cdc.sh
[2026-01-19 13:54:13]    Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  CDC Verification (User Contracts)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📊 Checking Debezium Connector: pg_user_sau_main_dev_user_contracts_debezium
   ✅ Debezium Connector: RUNNING
   📋 Task State: RUNNING

📊 Checking ES Sink Connector: pg_user_sau_main_dev_coordinator_user_contracts_es_sink
   ❌ ES Sink Connector: NOT RUNNING or NOT FOUND
   Response: {"error_code":404,"message":"No status found for connector pg_user_sau_main_dev_coordinator_user_contracts_es_sink"}

📊 Checking PostgreSQL Replication Slot...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
   ⚠️ Replication Slot Not Found: slot_user_sau_main_dev_user_contracts

📊 Checking PostgreSQL Publication...
   ⚠️ Publication Not Found: cdc_pub_user_contracts

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  CDC Verification Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:54:14] ✅ Completed: 05-verify-cdc.sh
[2026-01-19 13:54:14] 
[2026-01-19 13:54:14] ==========================================
[2026-01-19 13:54:14] ✅ CDC Pipeline setup complete for 1 subservice(s)
[2026-01-19 13:54:14] CDC SETUP SCRIPT FINISHED
[2026-01-19 13:54:14] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260119_134940.log
[2026-01-19 13:54:14] ==========================================
✓ ✅ CDC Pipeline setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 11-monitoring-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up monitoring for coordinator...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 PostgreSQL Monitoring Integration for user-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-user-sau-main-dev
[OK]   Observability cell endpoints registered for user-sau-main-dev
[OK]   ✓ Observability cell is ready

[INFO] ✓ Using private IP for metrics: 10.100.1.231
[INFO] 2️⃣ Setting up postgres_exporter integration...
[INFO] Checking observability cell readiness: obs-user-sau-main-dev
[OK]   Observability cell endpoints registered for user-sau-main-dev
[INFO] Setting up postgres_exporter for user-sau-main-dev
[2026-01-19 13:54:16 UTC] USER=www-data EUID=0 PID=1414851 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-user-sau-main-dev.yaml /etc/prometheus/postgres_exporter_queries-user-sau-main-dev.yaml
[2026-01-19 13:54:16 UTC] USER=www-data EUID=0 PID=1414860 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-user-sau-main-dev.yaml
[2026-01-19 13:54:16 UTC] USER=www-data EUID=0 PID=1414870 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-user-sau-main-dev.yaml
[OK]   Custom queries file created at /etc/prometheus/postgres_exporter_queries-user-sau-main-dev.yaml
[2026-01-19 13:54:16 UTC] USER=www-data EUID=0 PID=1414886 ACTION=passthru ARGS=mv /tmp/postgres_exporter-user-sau-main-dev.service /etc/systemd/system/postgres_exporter-user-sau-main-dev.service
[2026-01-19 13:54:16 UTC] USER=www-data EUID=0 PID=1414897 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-19 13:54:17 UTC] USER=www-data EUID=0 PID=1414946 ACTION=passthru ARGS=systemctl enable postgres_exporter-user-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/postgres_exporter-user-sau-main-dev.service → /etc/systemd/system/postgres_exporter-user-sau-main-dev.service.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  IP Conflict Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
IP Address:  10.100.1.231
Port:        9187
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔍 Checking IP conflict for user-sau-main-dev on 10.100.1.231:9187...
✅ IP 10.100.1.231:9187 is available - no conflicts detected

🔍 Checking for orphaned processes that might conflict...
✅ No orphaned processes detected

✅ All checks passed - safe to proceed with user-sau-main-dev setup
[2026-01-19 13:54:18 UTC] USER=www-data EUID=0 PID=1415044 ACTION=passthru ARGS=systemctl restart postgres_exporter-user-sau-main-dev.service
[OK]   postgres_exporter configured on db-user-sau-main-dev-postgresql.fastorder.com:9187
[INFO] Adding PostgreSQL scrape target to Prometheus config...
[OK]   PostgreSQL scrape target added
[INFO] Creating PostgreSQL alert rules...
[2026-01-19 13:54:20 UTC] USER=www-data EUID=0 PID=1415118 ACTION=fsop ARGS=mv /tmp/postgresql_alerts_user-sau-main-dev.yml /etc/prometheus/obs-user-sau-main-dev/rules/postgresql_alerts.yml
[OK]   PostgreSQL alert rules created: /etc/prometheus/obs-user-sau-main-dev/rules/postgresql_alerts.yml
[INFO] Adding PostgreSQL alerts to Prometheus config...
[2026-01-19 13:54:20 UTC] USER=www-data EUID=0 PID=1415128 ACTION=fsop ARGS=sed -i /rule_files:/a\  - "rules/postgresql_alerts.yml" /etc/prometheus/obs-user-sau-main-dev/prometheus.yml
[OK]   PostgreSQL alerts registered in Prometheus
[2026-01-19 13:54:20 UTC] USER=www-data EUID=0 PID=1415139 ACTION=passthru ARGS=systemctl reload prometheus-obs-user-sau-main-dev.service
Failed to reload prometheus-obs-user-sau-main-dev.service: Job type reload is not applicable for unit prometheus-obs-user-sau-main-dev.service.
[2026-01-19 13:54:20 UTC] USER=www-data EUID=0 PID=1415149 ACTION=passthru ARGS=systemctl restart prometheus-obs-user-sau-main-dev.service
[OK]   Prometheus reloaded with PostgreSQL monitoring
[OK]   ✓ postgres_exporter integration complete
[INFO] Registering postgres_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: postgres_exporter -> 10.100.1.231:9187
[OK]   ✓ Registered postgres_exporter scrape target: 10.100.1.231:9187
[INFO]   Target file: /etc/prometheus/obs-user-sau-main-dev/targets/postgres_exporter.yml
[OK]   ✓ postgres_exporter registered as Prometheus scrape target

[INFO] 3️⃣ Setting up pgbouncer_exporter integration...
[INFO] PgBouncer FQDN found in /etc/hosts: db-user-sau-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.184
[INFO] PgBouncer detected: db-user-sau-main-dev-postgresql-bouncer.fastorder.com:6432
[OK]   ✓ pgbouncer_exporter already installed
[INFO] Getting pgbouncer_admin password (SERVICE=user, ZONE=sau)
[OK]   ✓ pgbouncer_admin password retrieved (24 chars)
[INFO] Using pgbouncer certs from: /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
[INFO] Creating pgbouncer_exporter systemd service...
[OK]   ✓ pgbouncer_exporter service file created
[INFO] Starting pgbouncer_exporter service...
[2026-01-19 13:54:22 UTC] USER=www-data EUID=0 PID=1415226 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-19 13:54:22 UTC] USER=www-data EUID=0 PID=1415271 ACTION=passthru ARGS=systemctl enable pgbouncer_exporter-user-sau-main-dev.service
[2026-01-19 13:54:23 UTC] USER=www-data EUID=0 PID=1415319 ACTION=passthru ARGS=systemctl restart pgbouncer_exporter-user-sau-main-dev.service
[OK]   ✓ pgbouncer_exporter service running
[INFO] Registering pgbouncer_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: pgbouncer_exporter -> 10.100.1.184:9127
[OK]   ✓ Registered pgbouncer_exporter scrape target: 10.100.1.184:9127
[INFO]   Target file: /etc/prometheus/obs-user-sau-main-dev/targets/pgbouncer_exporter.yml
[OK]   ✓ pgbouncer_exporter registered as Prometheus scrape target

[INFO] 4️⃣ Registering nodes to monitoring database...
[INFO] PostgreSQL key permissions set for www-data access: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key
[INFO] Registering PostgreSQL coordinator to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        user-sau-main-dev-postgresql-coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.231
[INFO]   Port:              5432
[INFO]   FQDN:              db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 83d5bc7d-3699-4f7e-98b2-72fdfea60e05
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[OK]   ✓ PostgreSQL coordinator registered
[INFO] Registering PgBouncer to monitoring dashboard...
[INFO]   FQDN: db-user-sau-main-dev-postgresql-bouncer.fastorder.com, IP: 10.100.1.184, Port: 6432
[INFO]   Key permissions set for www-data access
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        user-sau-main-dev-pgbouncer
[INFO]   Identifier Parent: pooling
[INFO]   IP:                10.100.1.184
[INFO]   Port:              6432
[INFO]   FQDN:              db-user-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: c866fe26-0c2d-4619-a98d-8cd82c922b78
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[OK]   ✓ PgBouncer registered

[INFO] 5️⃣ Creating PgBouncer professional monitoring rules...
[INFO] Creating PgBouncer recording rules...
[OK]   ✓ PgBouncer recording rules created
[INFO] Creating PgBouncer alert rules with runbook URLs...
[OK]   ✓ PgBouncer alert rules with runbook URLs created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] 6️⃣ Setting up pgbackrest_exporter integration...
[INFO] pgBackRest detected, setting up exporter...
[OK]   ✓ pgbackrest_exporter already installed
[INFO] Creating pgbackrest_exporter systemd service...
[OK]   ✓ pgbackrest_exporter service file created
[INFO] Starting pgbackrest_exporter service...
[2026-01-19 13:54:27 UTC] USER=www-data EUID=0 PID=1415515 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-19 13:54:28 UTC] USER=www-data EUID=0 PID=1415562 ACTION=passthru ARGS=systemctl enable pgbackrest_exporter-user-sau-main-dev.service
[2026-01-19 13:54:28 UTC] USER=www-data EUID=0 PID=1415630 ACTION=passthru ARGS=systemctl restart pgbackrest_exporter-user-sau-main-dev.service
[WARN] ⚠️  pgbackrest_exporter service not running (may need manual start)
[WARN]     Run: systemctl status pgbackrest_exporter-user-sau-main-dev.service
[INFO] Creating pgBackRest alert rules...
[OK]   ✓ pgBackRest alert rules created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ PostgreSQL & PgBouncer Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] postgres_exporter: http://localhost:9187/metrics
[INFO] pgbouncer_exporter: http://localhost:9127/metrics
[INFO] Prometheus: https://metrics-user-sau-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-user-sau-main-dev.fastorder.com
[INFO] 
[INFO] PgBouncer Monitoring:
[INFO]   • Recording rules: /etc/prometheus/obs-user-sau-main-dev/rules/pgbouncer_recording_rules.yml
[INFO]   • Alert rules: /etc/prometheus/obs-user-sau-main-dev/rules/pgbouncer_alerts.yml
[INFO] 
[INFO] pgBackRest Monitoring:
[INFO]   • Alert rules: /etc/prometheus/obs-user-sau-main-dev/rules/pgbackrest_alerts.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up monitoring for 1 worker(s) and 1 standby(s) per worker...
[INFO] Setting up monitoring for: worker-01
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 PostgreSQL Monitoring Integration for user-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-user-sau-main-dev
[OK]   Observability cell endpoints registered for user-sau-main-dev
[OK]   ✓ Observability cell is ready

[INFO] ✓ Using private IP for metrics: 10.100.1.231
[INFO] 2️⃣ Setting up postgres_exporter integration...
[INFO] Checking observability cell readiness: obs-user-sau-main-dev
[OK]   Observability cell endpoints registered for user-sau-main-dev
[INFO] Setting up postgres_exporter for user-sau-main-dev
[2026-01-19 13:54:31 UTC] USER=www-data EUID=0 PID=1415813 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-user-sau-main-dev.yaml /etc/prometheus/postgres_exporter_queries-user-sau-main-dev.yaml
[2026-01-19 13:54:32 UTC] USER=www-data EUID=0 PID=1415822 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-user-sau-main-dev.yaml
[2026-01-19 13:54:32 UTC] USER=www-data EUID=0 PID=1415831 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-user-sau-main-dev.yaml
[OK]   Custom queries file created at /etc/prometheus/postgres_exporter_queries-user-sau-main-dev.yaml
[OK]   postgres_exporter already running with custom queries for user-sau-main-dev
[OK]   ✓ postgres_exporter integration complete
[INFO] Registering postgres_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: postgres_exporter -> 10.100.1.231:9187
[OK]   ✓ Registered postgres_exporter scrape target: 10.100.1.231:9187
[INFO]   Target file: /etc/prometheus/obs-user-sau-main-dev/targets/postgres_exporter.yml
[OK]   ✓ postgres_exporter registered as Prometheus scrape target

[INFO] 3️⃣ Setting up pgbouncer_exporter integration...
[INFO] PgBouncer FQDN found in /etc/hosts: db-user-sau-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.184
[INFO] PgBouncer detected: db-user-sau-main-dev-postgresql-bouncer.fastorder.com:6432
[OK]   ✓ pgbouncer_exporter already installed
[INFO] Getting pgbouncer_admin password (SERVICE=user, ZONE=sau)
[OK]   ✓ pgbouncer_admin password retrieved (24 chars)
[INFO] Using pgbouncer certs from: /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
[INFO] Creating pgbouncer_exporter systemd service...
[OK]   ✓ pgbouncer_exporter service file created
[INFO] Starting pgbouncer_exporter service...
[2026-01-19 13:54:33 UTC] USER=www-data EUID=0 PID=1415901 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-19 13:54:34 UTC] USER=www-data EUID=0 PID=1415952 ACTION=passthru ARGS=systemctl enable pgbouncer_exporter-user-sau-main-dev.service
[2026-01-19 13:54:34 UTC] USER=www-data EUID=0 PID=1416004 ACTION=passthru ARGS=systemctl restart pgbouncer_exporter-user-sau-main-dev.service
[OK]   ✓ pgbouncer_exporter service running
[INFO] Registering pgbouncer_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: pgbouncer_exporter -> 10.100.1.184:9127
[OK]   ✓ Registered pgbouncer_exporter scrape target: 10.100.1.184:9127
[INFO]   Target file: /etc/prometheus/obs-user-sau-main-dev/targets/pgbouncer_exporter.yml
[OK]   ✓ pgbouncer_exporter registered as Prometheus scrape target

[INFO] 4️⃣ Registering nodes to monitoring database...
[INFO] PostgreSQL key permissions set for www-data access: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[INFO] Registering PostgreSQL worker-01 to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        user-sau-main-dev-postgresql-worker-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.231
[INFO]   Port:              5432
[INFO]   FQDN:              db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 87ccba48-d8e0-43e4-97b8-d87917a5d35c
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[OK]   ✓ PostgreSQL worker-01 registered
[INFO] Registering PgBouncer to monitoring dashboard...
[INFO]   FQDN: db-user-sau-main-dev-postgresql-bouncer.fastorder.com, IP: 10.100.1.184, Port: 6432
[INFO]   Key permissions set for www-data access
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        user-sau-main-dev-pgbouncer
[INFO]   Identifier Parent: pooling
[INFO]   IP:                10.100.1.184
[INFO]   Port:              6432
[INFO]   FQDN:              db-user-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: c866fe26-0c2d-4619-a98d-8cd82c922b78
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[OK]   ✓ PgBouncer registered

[INFO] 5️⃣ Creating PgBouncer professional monitoring rules...
[INFO] Creating PgBouncer recording rules...
[OK]   ✓ PgBouncer recording rules created
[INFO] Creating PgBouncer alert rules with runbook URLs...
[OK]   ✓ PgBouncer alert rules with runbook URLs created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] 6️⃣ Setting up pgbackrest_exporter integration...
[INFO] pgBackRest detected, setting up exporter...
[OK]   ✓ pgbackrest_exporter already installed
[INFO] Creating pgbackrest_exporter systemd service...
[OK]   ✓ pgbackrest_exporter service file created
[INFO] Starting pgbackrest_exporter service...
[2026-01-19 13:54:38 UTC] USER=www-data EUID=0 PID=1416243 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-19 13:54:39 UTC] USER=www-data EUID=0 PID=1416289 ACTION=passthru ARGS=systemctl enable pgbackrest_exporter-user-sau-main-dev.service
[2026-01-19 13:54:39 UTC] USER=www-data EUID=0 PID=1416335 ACTION=passthru ARGS=systemctl restart pgbackrest_exporter-user-sau-main-dev.service
[WARN] ⚠️  pgbackrest_exporter service not running (may need manual start)
[WARN]     Run: systemctl status pgbackrest_exporter-user-sau-main-dev.service
[INFO] Creating pgBackRest alert rules...
[OK]   ✓ pgBackRest alert rules created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ PostgreSQL & PgBouncer Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] postgres_exporter: http://localhost:9187/metrics
[INFO] pgbouncer_exporter: http://localhost:9127/metrics
[INFO] Prometheus: https://metrics-user-sau-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-user-sau-main-dev.fastorder.com
[INFO] 
[INFO] PgBouncer Monitoring:
[INFO]   • Recording rules: /etc/prometheus/obs-user-sau-main-dev/rules/pgbouncer_recording_rules.yml
[INFO]   • Alert rules: /etc/prometheus/obs-user-sau-main-dev/rules/pgbouncer_alerts.yml
[INFO] 
[INFO] pgBackRest Monitoring:
[INFO]   • Alert rules: /etc/prometheus/obs-user-sau-main-dev/rules/pgbackrest_alerts.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up monitoring for standby: worker-01-standby-01
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 PostgreSQL Monitoring Integration for user-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-user-sau-main-dev
[OK]   Observability cell endpoints registered for user-sau-main-dev
[OK]   ✓ Observability cell is ready

[INFO] ✓ Using private IP for metrics: 10.100.1.231
[INFO] 2️⃣ Setting up postgres_exporter integration...
[INFO] Checking observability cell readiness: obs-user-sau-main-dev
[OK]   Observability cell endpoints registered for user-sau-main-dev
[INFO] Setting up postgres_exporter for user-sau-main-dev
[2026-01-19 13:54:43 UTC] USER=www-data EUID=0 PID=1416569 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-user-sau-main-dev.yaml /etc/prometheus/postgres_exporter_queries-user-sau-main-dev.yaml
[2026-01-19 13:54:43 UTC] USER=www-data EUID=0 PID=1416578 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-user-sau-main-dev.yaml
[2026-01-19 13:54:43 UTC] USER=www-data EUID=0 PID=1416587 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-user-sau-main-dev.yaml
[OK]   Custom queries file created at /etc/prometheus/postgres_exporter_queries-user-sau-main-dev.yaml
[OK]   postgres_exporter already running with custom queries for user-sau-main-dev
[OK]   ✓ postgres_exporter integration complete
[INFO] Registering postgres_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: postgres_exporter -> 10.100.1.231:9187
[OK]   ✓ Registered postgres_exporter scrape target: 10.100.1.231:9187
[INFO]   Target file: /etc/prometheus/obs-user-sau-main-dev/targets/postgres_exporter.yml
[OK]   ✓ postgres_exporter registered as Prometheus scrape target

[INFO] 3️⃣ Setting up pgbouncer_exporter integration...
[INFO] PgBouncer FQDN found in /etc/hosts: db-user-sau-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.184
[INFO] PgBouncer detected: db-user-sau-main-dev-postgresql-bouncer.fastorder.com:6432
[OK]   ✓ pgbouncer_exporter already installed
[INFO] Getting pgbouncer_admin password (SERVICE=user, ZONE=sau)
[OK]   ✓ pgbouncer_admin password retrieved (24 chars)
[INFO] Using pgbouncer certs from: /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
[INFO] Creating pgbouncer_exporter systemd service...
[OK]   ✓ pgbouncer_exporter service file created
[INFO] Starting pgbouncer_exporter service...
[2026-01-19 13:54:45 UTC] USER=www-data EUID=0 PID=1416679 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-19 13:54:45 UTC] USER=www-data EUID=0 PID=1416727 ACTION=passthru ARGS=systemctl enable pgbouncer_exporter-user-sau-main-dev.service
[2026-01-19 13:54:46 UTC] USER=www-data EUID=0 PID=1416776 ACTION=passthru ARGS=systemctl restart pgbouncer_exporter-user-sau-main-dev.service
[OK]   ✓ pgbouncer_exporter service running
[INFO] Registering pgbouncer_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: pgbouncer_exporter -> 10.100.1.184:9127
[OK]   ✓ Registered pgbouncer_exporter scrape target: 10.100.1.184:9127
[INFO]   Target file: /etc/prometheus/obs-user-sau-main-dev/targets/pgbouncer_exporter.yml
[OK]   ✓ pgbouncer_exporter registered as Prometheus scrape target

[INFO] 4️⃣ Registering nodes to monitoring database...
[INFO] Registering PostgreSQL worker-01-standby-01 to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        user-sau-main-dev-postgresql-worker-01-standby-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.231
[INFO]   Port:              5432
[INFO]   FQDN:              db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 6b53354f-af0c-46ce-9112-1ad9eae0ff4a
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[OK]   ✓ PostgreSQL worker-01-standby-01 registered
[INFO] Registering PgBouncer to monitoring dashboard...
[INFO]   FQDN: db-user-sau-main-dev-postgresql-bouncer.fastorder.com, IP: 10.100.1.184, Port: 6432
[INFO]   Key permissions set for www-data access
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        user-sau-main-dev-pgbouncer
[INFO]   Identifier Parent: pooling
[INFO]   IP:                10.100.1.184
[INFO]   Port:              6432
[INFO]   FQDN:              db-user-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: c866fe26-0c2d-4619-a98d-8cd82c922b78
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[OK]   ✓ PgBouncer registered

[INFO] 5️⃣ Creating PgBouncer professional monitoring rules...
[INFO] Creating PgBouncer recording rules...
[OK]   ✓ PgBouncer recording rules created
[INFO] Creating PgBouncer alert rules with runbook URLs...
[OK]   ✓ PgBouncer alert rules with runbook URLs created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] 6️⃣ Setting up pgbackrest_exporter integration...
[INFO] pgBackRest detected, setting up exporter...
[OK]   ✓ pgbackrest_exporter already installed
[INFO] Creating pgbackrest_exporter systemd service...
[OK]   ✓ pgbackrest_exporter service file created
[INFO] Starting pgbackrest_exporter service...
[2026-01-19 13:54:50 UTC] USER=www-data EUID=0 PID=1416968 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-19 13:54:51 UTC] USER=www-data EUID=0 PID=1417014 ACTION=passthru ARGS=systemctl enable pgbackrest_exporter-user-sau-main-dev.service
[2026-01-19 13:54:51 UTC] USER=www-data EUID=0 PID=1417059 ACTION=passthru ARGS=systemctl restart pgbackrest_exporter-user-sau-main-dev.service
[WARN] ⚠️  pgbackrest_exporter service not running (may need manual start)
[WARN]     Run: systemctl status pgbackrest_exporter-user-sau-main-dev.service
[INFO] Creating pgBackRest alert rules...
[OK]   ✓ pgBackRest alert rules created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ PostgreSQL & PgBouncer Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] postgres_exporter: http://localhost:9187/metrics
[INFO] pgbouncer_exporter: http://localhost:9127/metrics
[INFO] Prometheus: https://metrics-user-sau-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-user-sau-main-dev.fastorder.com
[INFO] 
[INFO] PgBouncer Monitoring:
[INFO]   • Recording rules: /etc/prometheus/obs-user-sau-main-dev/rules/pgbouncer_recording_rules.yml
[INFO]   • Alert rules: /etc/prometheus/obs-user-sau-main-dev/rules/pgbouncer_alerts.yml
[INFO] 
[INFO] pgBackRest Monitoring:
[INFO]   • Alert rules: /etc/prometheus/obs-user-sau-main-dev/rules/pgbackrest_alerts.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ Monitoring setup completed for coordinator, workers, and standbys

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 12-setup-offsite-backup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Setting up offsite backup repository for user-sau-main-dev...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Offsite Backup Repository Setup (repo2)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📋 OFFSITE BACKUP INFORMATION
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Why Offsite Backups?
[INFO]   ✓ Disaster recovery resilience (datacenter loss, hardware failure)
[INFO]   ✓ Protection against local corruption or ransomware
[INFO]   ✓ Compliance requirements (geographic redundancy)
[INFO]   ✓ Long-term archival with cost-effective storage tiers

[WARN] ⚠️  Offsite backup (repo2) is NOT ENABLED
[WARN]    Using local backups only (repo1)

[INFO] Configuration Example Location:
[INFO]   📄 /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example

[INFO] Supported Storage Backends:
[INFO]   • AWS S3 (standard, multi-region)
[INFO]   • AWS S3 Glacier (low-cost archival)
[INFO]   • MinIO (self-hosted S3-compatible)
[INFO]   • Google Cloud Storage (via S3 compatibility)
[INFO]   • Azure Blob Storage (via S3 compatibility)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📝 SETUP INSTRUCTIONS
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Step 1: Review the example configuration
[INFO]   cat /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example

[INFO] Step 2: Prepare S3 bucket and credentials
[INFO]   • Create S3 bucket (or MinIO bucket)
[INFO]   • Create IAM user with S3 permissions (PutObject, GetObject, DeleteObject, ListBucket)
[INFO]   • Note: Access Key ID and Secret Access Key

[INFO] Step 3: Add repo2 configuration to /etc/pgbackrest/pgbackrest.conf
[INFO]   • Copy repo2-* settings from example to [global] section
[INFO]   • Replace placeholders (bucket name, access keys, region)
[INFO]   • Note: Use same cipher key as repo1, or generate separate key for repo2

[INFO] Step 4: Initialize repo2 stanzas
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=user-sau-main-dev-coordinator stanza-create --repo=2
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=user-sau-main-dev-worker-01 stanza-create --repo=2
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=user-sau-main-dev-worker-02 stanza-create --repo=2

[INFO] Step 5: Verify repo2 configuration
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=user-sau-main-dev-coordinator check --repo=2

[INFO] Step 6: Take initial full backup to repo2
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=user-sau-main-dev-coordinator --repo=2 --type=full backup

[INFO] Step 7: Update backup automation to include repo2
[INFO]   • Edit: /usr/local/bin/pgbackrest-full-backup-user-sau-main-dev.sh
[INFO]   • Change: pgbackrest backup to pgbackrest --repo=1,2 backup
[INFO]   • Or: Add separate cron for repo2 backups

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🧪 TESTING
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] After configuration, run:
[INFO]   ./08-setup-offsite-backup.sh test

[INFO] This will verify:
[INFO]   ✓ S3 connectivity
[INFO]   ✓ Stanza initialization
[INFO]   ✓ Test backup and restore from repo2

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 💡 COST OPTIMIZATION
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] AWS S3 Lifecycle Policies (transition to cheaper storage):
[INFO]   • 0-30 days:   S3 Standard (~$0.023/GB/month)
[INFO]   • 30-90 days:  S3 Standard-IA (~$0.0125/GB/month)
[INFO]   • 90+ days:    S3 Glacier (~$0.004/GB/month)

[INFO] Estimated costs for 100GB backups:
[INFO]   • All Standard:     ~$2.30/month
[INFO]   • With lifecycle:   ~$1.20/month (48% savings)


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 13-setup-monitoring-alerts.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Setting up backup monitoring and alerting for user-sau-main-dev...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Monitoring and Alerting Configuration
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] This will set up monitoring for:
  • Backup failures (cron job failures)
  • WAL archiving backlog (>100 files)
  • Repository disk space (<20% free)
  • Backup age (>25 hours)

[INFO] No alert email configured (set ALERT_EMAIL environment variable)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Creating monitoring directories...
[2026-01-19 13:54:58 UTC] USER=www-data EUID=0 PID=1417307 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest-monitoring
[2026-01-19 13:54:58 UTC] USER=www-data EUID=0 PID=1417344 ACTION=fsop ARGS=chmod 777 /opt/pgbackrest-monitoring
[2026-01-19 13:54:58 UTC] USER=www-data EUID=0 PID=1417366 ACTION=fsop ARGS=chmod 777 /var/log/pgbackrest-monitoring
[2026-01-19 13:54:58 UTC] USER=www-data EUID=0 PID=1417405 ACTION=fsop ARGS=chown postgres:postgres /opt/pgbackrest-monitoring
[2026-01-19 13:54:58 UTC] USER=www-data EUID=0 PID=1417431 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest-monitoring
[INFO] ✅ Directories created

[INFO] 2️⃣ Creating alert helper script...
[2026-01-19 13:54:59 UTC] USER=www-data EUID=0 PID=1417463 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/send-alert.sh
[INFO] ✅ Alert helper created

[INFO] 3️⃣ Creating WAL queue monitoring script...
[2026-01-19 13:54:59 UTC] USER=www-data EUID=0 PID=1417483 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-wal-queue.sh
[INFO] ✅ WAL queue monitor created

[INFO] 4️⃣ Creating backup age monitoring script...
[INFO] ✅ Backup age monitor created

[INFO] 5️⃣ Creating repository disk space monitoring script...
[2026-01-19 13:54:59 UTC] USER=www-data EUID=0 PID=1417537 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-repo-space.sh
[INFO] ✅ Disk space monitor created

[INFO] 6️⃣ Creating backup failure detection script...
[2026-01-19 13:54:59 UTC] USER=www-data EUID=0 PID=1417557 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-backup-failures.sh
[INFO] ✅ Backup failure detector created

[INFO] 7️⃣ Creating master monitoring script...
[2026-01-19 13:54:59 UTC] USER=www-data EUID=0 PID=1417580 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO] ✅ Master monitoring script created

[INFO] 8️⃣ Installing mailutils for email alerts...
[INFO] ✅ mailutils already installed

[INFO] 9️⃣ Installing jq for JSON parsing...
[INFO] ✅ jq already installed

[INFO] 🔟 Setting up monitoring cron jobs...
[2026-01-19 13:54:59 UTC] USER=www-data EUID=0 PID=1417602 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-monitoring-user-sau-main-dev
[INFO] ✅ Monitoring cron jobs configured
[INFO]    Checks run every 15 minutes

[INFO] 1️⃣1️⃣ Creating monitoring dashboard...
[2026-01-19 13:54:59 UTC] USER=www-data EUID=0 PID=1417622 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/dashboard.sh
[INFO] ✅ Monitoring dashboard created

[INFO] 1️⃣2️⃣ Running initial monitoring check...

[2026-01-19 13:54:59 UTC] USER=www-data EUID=0 PID=1417631 ACTION=passthru ARGS=bash /opt/pgbackrest-monitoring/run-all-checks.sh

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup monitoring setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Monitoring Configuration:
[INFO]   Alert Email:        
[INFO]   Slack Webhook:      Not configured

[INFO] Monitoring Checks:
[INFO]   • WAL Queue:        Every 15 minutes (threshold: >100 files)
[INFO]   • Backup Age:       Every 15 minutes (threshold: >25 hours)
[INFO]   • Disk Space:       Every 15 minutes (threshold: <20% free)
[INFO]   • Backup Failures:  Every 15 minutes (log analysis)

[INFO] Scripts Created:
[INFO]   Monitoring dir:     /opt/pgbackrest-monitoring
[INFO]   Log dir:            /var/log/pgbackrest-monitoring
[INFO]   Dashboard:          /opt/pgbackrest-monitoring/dashboard.sh
[INFO]   Master check:       /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO]   Alert sender:       /opt/pgbackrest-monitoring/send-alert.sh

[INFO] Useful Commands:
[INFO]   View dashboard:     /usr/local/bin/fastorder-provisioning-wrapper.sh /opt/pgbackrest-monitoring/dashboard.sh
[INFO]   Run checks now:     /usr/local/bin/fastorder-provisioning-wrapper.sh /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO]   View alerts:        tail -f /var/log/pgbackrest-monitoring/alerts.log
[INFO]   View monitoring:    tail -f /var/log/pgbackrest-monitoring/monitoring.log

[INFO] Cron Schedule:
[INFO]   All checks:         Every 15 minutes
[INFO]   Log rotation:       Weekly (keep 7 days)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 14-vault-cipher-key.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] ✅ Using permanent AWS credentials from /home/ab/.aws/credentials [default] profile
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔐 PostgreSQL Cipher Key Vaulting
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Environment:       user-sau-main-dev
[INFO]   AWS Region:        me-central-1
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣  Verifying AWS setup...
[INFO] ✅ AWS authentication successful

[INFO] 2️⃣  Verifying cipher key...
[INFO] ✅ Cipher key found
[INFO]    Location: /etc/pgbackrest/.cipher-key-user-sau-main-dev
[INFO]    Hash (MD5): 71c290b5e73fca0bc8d9f07b411e1a69
[INFO]    Size: 188 bytes

[INFO] 3️⃣  Vaulting cipher key to AWS Secrets Manager...
[INFO]    Secret name: fastorder/db/user/sau/main/dev/postgresql/pgbackrest/cipher-key
[INFO]    Secret exists, updating value...
[INFO] ✅ Cipher key updated in AWS Secrets Manager
[INFO]    Verifying storage...
[INFO] ✅ Verification successful - key matches

[INFO] 4️⃣  Creating local encrypted backup...
[2026-01-19 13:55:09 UTC] USER=www-data EUID=0 PID=1418057 ACTION=fsop ARGS=mv /tmp/cipher-key-backup-1417822.enc /root/.pgbackrest-cipher-key-user-sau-main-dev.enc
[2026-01-19 13:55:09 UTC] USER=www-data EUID=0 PID=1418066 ACTION=fsop ARGS=chmod 600 /root/.pgbackrest-cipher-key-user-sau-main-dev.enc
[2026-01-19 13:55:09 UTC] USER=www-data EUID=0 PID=1418085 ACTION=fsop ARGS=chmod 600 /root/.pgbackrest-cipher-key-passphrase-user-sau-main-dev.txt
[INFO] ✅ Local encrypted backup created
[INFO]    Backup file: /root/.pgbackrest-cipher-key-user-sau-main-dev.enc
[INFO]    Passphrase: /root/.pgbackrest-cipher-key-passphrase-user-sau-main-dev.txt

[INFO] 5️⃣  Vaulting backup passphrase...
[INFO] ✅ Backup passphrase updated

[INFO] 6️⃣  Creating recovery documentation...
[2026-01-19 13:55:12 UTC] USER=www-data EUID=0 PID=1418163 ACTION=fsop ARGS=chmod 640 /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_user-sau-main-dev.md
[2026-01-19 13:55:12 UTC] USER=www-data EUID=0 PID=1418174 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_user-sau-main-dev.md
[INFO] ✅ Recovery documentation: /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_user-sau-main-dev.md

[INFO] 7️⃣  Storing backup metadata...
[INFO] ✅ Backup metadata stored in AWS Secrets Manager
[INFO]    Secret: fastorder/db/user/sau/main/dev/postgresql/backup/metadata-20260119-135512

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Cipher Key Vaulting Complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO]   Environment:          user-sau-main-dev
[INFO]   Key Hash:             71c290b5e73fca0bc8d9f07b411e1a69

[INFO] AWS Secrets:
[INFO]   Cipher Key:           fastorder/db/user/sau/main/dev/postgresql/pgbackrest/cipher-key
[INFO]   Passphrase:           fastorder/db/user/sau/main/dev/postgresql/pgbackrest/cipher-key-passphrase
[INFO]   Backup Metadata:      fastorder/db/user/sau/main/dev/postgresql/backup/metadata-20260119-135512

[INFO] Local Backups:
[INFO]   Encrypted File:       /root/.pgbackrest-cipher-key-user-sau-main-dev.enc
[INFO]   Passphrase File:      /root/.pgbackrest-cipher-key-passphrase-user-sau-main-dev.txt

[INFO] Recovery Doc:           /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_user-sau-main-dev.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Skipping 15-backup-restore-test.sh (test script - set RUN_TESTS=true to enable)
[INFO] Skipping 16-test-recovery.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 17-verification.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)

[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] PostgreSQL Production Readiness Verification
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] 
[INFO] This script verifies 3 CRITICAL checks for production readiness:
[INFO]   1. Citus Cluster Operational (coordinator + workers)
[INFO]   2. SSL/TLS Enforced (certificates valid, connections secure)
[INFO]   3. Coordinator Backups Configured (pgBackRest functional)
[INFO] 
[INFO] 📖 Documentation: /tmp/VERIFICATION_RUNBOOK.md
[INFO] 🔐 Security: Uses sudo for certificate checks (maintains strict permissions)
[INFO] 📊 Exit Code: 0 = production ready, 1 = critical checks failed
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🕐 Ensuring all PostgreSQL services are ready...
[ OK ] ✅ All PostgreSQL services are ready

[INFO] 🔍 Starting PostgreSQL verification...
[INFO] Environment: user-sau-main-dev
[INFO] Citus: yes

[INFO] Citus mode ENABLED
[INFO] → Coordinator + 1 worker(s) + 3 HA node(s) per worker

[INFO] Verifying 1 worker(s)...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying: worker-01 (type: worker-01)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Starting PostgreSQL verification for user-sau-main-dev-worker-01...

[INFO] 1️⃣ Checking systemd service status...
[OK]   ✅ Service postgresql@user-sau-main-dev-worker-01.service is active

[INFO] 2️⃣ Checking PostgreSQL process...
[OK]   ✅ PostgreSQL process is running

[INFO] 3️⃣ Checking socket directory...
[OK]   ✅ Socket directory exists: /var/run/postgresql-user-sau-main-dev-worker-01
total 4
drwxrwsr-x  2 postgres postgres   80 Jan 19 13:44 .
drwxr-xr-x 61 root     root     1600 Jan 19 13:55 ..
srwxrwxrwx  1 postgres postgres    0 Jan 19 13:44 .s.PGSQL.5432
-rw-------  1 postgres postgres  120 Jan 19 13:44 .s.PGSQL.5432.lock

[INFO] 4️⃣ Testing connection via Unix socket...
[OK]   ✅ Socket connection successful
                                                              version                                                              
-----------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.6 (Ubuntu 17.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

[INFO] 5️⃣ Checking SSL certificates...
[2026-01-19 13:55:16 UTC] USER=www-data EUID=0 PID=1418425 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[OK]   ✅ Server certificate exists: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-19 13:55:16 UTC] USER=www-data EUID=0 PID=1418436 ACTION=fsop ARGS=openssl x509 -in /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt -noout -checkend 86400
Certificate will not expire
[OK]   ✅ Server certificate is valid
[2026-01-19 13:55:16 UTC] USER=www-data EUID=0 PID=1418445 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[OK]   ✅ CA certificate exists: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[INFO] ℹ️  Client certificates not found at /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[INFO]     (This is OK if using password authentication)

[INFO] 6️⃣ Checking PostgreSQL settings...
[OK]   ✅ SSL is enabled worker-01 worker-01
[OK]   ✅ Max connections: 100
[OK]   ✅ Listen addresses: 10.100.1.232
[OK]   ✅ WAL level: logical
[OK]   ✅ Shared preload libraries: shared_preload_libraries

[INFO] 7️⃣ Checking replication configuration...
[INFO] ℹ️  No synchronous standbys configured (single node or async replication)
[INFO] Checking replication slots...
      slot_name       | slot_type | active | restart_lsn 
----------------------+-----------+--------+-------------
 worker_01_standby_01 | physical  | f      | 
(1 row)
[OK]   ✅ Replication slot naming uses underscores (correct)
[INFO] Checking active replication connections...
 application_name | client_addr | state | sync_state 
------------------+-------------+-------+------------
(0 rows)
[INFO] ℹ️  No active replication connections
[INFO] ℹ️  This is a PRIMARY node (no standby.signal)

[INFO] 8️⃣ Checking pg_hba.conf for replication rules...
[WARN] ⚠️ pg_hba.conf not found at /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf

[INFO] 9️⃣ Checking Citus configuration...
[OK]   ✅ Citus extension is installed
[OK]   ✅ Citus version: Citus 13.2.0
[OK]   ✅ max_prepared_transactions: 100 (adequate for Citus)
[INFO] Citus active worker nodes:
                        node_name                        | node_port 
---------------------------------------------------------+-----------
 db-user-sau-main-dev-postgresql-worker-01.fastorder.com |      5432
(1 row)



[INFO] 🔟 Checking data directory...
[OK]   ✅ Data directory exists: /data/postgresql/17/user-sau-main-dev/worker-01
[OK]   ✅ Data directory size: 4.0K

[INFO] 1️⃣1️⃣ Checking PgBouncer configuration...
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini' as root on web-03.
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/%i/pgbouncer.ini' as root on web-03.
[OK]   ✅ PgBouncer is installed
[INFO]    Version: 1.24.1
2.1.12-stable
c-ares
OpenSSL
yes
[OK]   ✅ PgBouncer service is active: pgbouncer@user-sau-main-dev.service
[WARN] ⚠️ PgBouncer IP service is not active: pgbouncer-ip@user-sau-main-dev.service
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini' as root on web-03.
[WARN] ⚠️ PgBouncer config not found: /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/user-sau-main-dev/userlist.txt' as root on web-03.
[WARN] ⚠️ PgBouncer auth file not found: /etc/pgbouncer/user-sau-main-dev/userlist.txt
[OK]   ✅ PgBouncer is listening on port 6432

[INFO] 1️⃣2️⃣ Enhanced PgBouncer Admin Console Verification...
[INFO] ℹ️  PgBouncer password not found

[INFO] 1️⃣3️⃣ Replicator User Connection Verification...
[INFO] Found 1 replication slot(s) - verifying replicator connectivity...
[WARN] ⚠️ Replicator certificates not found at /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[INFO]    Expected files:
[INFO]    - /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[INFO]    - /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[INFO]    - /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[INFO] Checking pg_hba.conf replicator rules...
[OK]   ✅ Replicator HBA rules found:
 line_number |  type   |   database    |  user_name   |   address    |  auth_method  | options | error 
-------------+---------+---------------+--------------+--------------+---------------+---------+-------
          20 | hostssl | {replication} | {replicator} | 10.100.1.233 | scram-sha-256 |         | 
          21 | hostssl | {replication} | {replicator} | 10.100.1.232 | scram-sha-256 |         | 
(2 rows)
[INFO] Checking active replicator connections in pg_stat_activity...
[WARN] ⚠️ No active replicator connections in pg_stat_activity
[WARN]    This is expected if standbys are not currently connected

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ PostgreSQL verification completed successfully!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Instance:       user-sau-main-dev-worker-01
[INFO] Service:        postgresql@user-sau-main-dev-worker-01.service
[INFO] Socket:         /var/run/postgresql-user-sau-main-dev-worker-01
[INFO] Data Directory: /data/postgresql/17/user-sau-main-dev/worker-01
[INFO] Hostname:       db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO] Port:           5432
[INFO] SSL:            on
[INFO] WAL Level:      logical
[INFO] Citus:          yes
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 💡 OPTIMIZATION OPPORTUNITIES (Optional Enhancements)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1. Review connection limits for production workload
[INFO]    🔌 Current: max_connections = 100 (PostgreSQL default)
[INFO]    💡 Consider: Increasing to 200-500 for production applications
[INFO]    ⚙️  Alternative: Use PgBouncer connection pooling (lower PostgreSQL limit, higher client capacity)
[INFO]    🔧 Action: Adjust max_connections in postgresql.conf based on workload analysis
[INFO]    ⚠️  Note: Each connection consumes ~10MB RAM; tune based on available memory
[INFO]    📚 Docs: https://www.postgresql.org/docs/current/runtime-config-connection.html

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ℹ️  These are optional enhancements for production-scale deployments
[INFO] ℹ️  Current configuration is fully functional and ready for production
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] ✅ Verification passed for worker-01

Failed to print table: Broken pipe
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying: worker-01-standby-01 (type: worker-01)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Starting PostgreSQL verification for user-sau-main-dev-worker-01-standby-01...

[INFO] 1️⃣ Checking systemd service status...
[OK]   ✅ Service postgresql@user-sau-main-dev-worker-01-standby-01.service is active

[INFO] 2️⃣ Checking PostgreSQL process...
[OK]   ✅ PostgreSQL process is running

[INFO] 3️⃣ Checking socket directory...
[OK]   ✅ Socket directory exists: /var/run/postgresql-user-sau-main-dev-worker-01-standby-01
total 4
drwxrwsr-x  2 postgres postgres   80 Jan 19 13:46 .
drwxr-xr-x 62 root     root     1620 Jan 19 13:55 ..
srwxrwxrwx  1 postgres postgres    0 Jan 19 13:46 .s.PGSQL.5432
-rw-------  1 postgres postgres  142 Jan 19 13:46 .s.PGSQL.5432.lock

[INFO] 4️⃣ Testing connection via Unix socket...
[OK]   ✅ Socket connection successful
                                                              version                                                              
-----------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.6 (Ubuntu 17.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

[INFO] 5️⃣ Checking SSL certificates...
[2026-01-19 13:55:29 UTC] USER=www-data EUID=0 PID=1419102 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[OK]   ✅ Server certificate exists: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[OK]   ✅ Server certificate is valid
[OK]   ✅ CA certificate exists: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt
[INFO] ℹ️  Client certificates not found at /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[INFO]     (This is OK if using password authentication)

[INFO] 6️⃣ Checking PostgreSQL settings...
[OK]   ✅ SSL is enabled worker-01 worker-01-standby-01
[OK]   ✅ Max connections: 100
[OK]   ✅ Listen addresses: 01
unknown
[OK]   ✅ WAL level: logical
[OK]   ✅ Shared preload libraries: shared_preload_libraries

[INFO] 7️⃣ Checking replication configuration...
[INFO] ℹ️  No synchronous standbys configured (single node or async replication)
[INFO] Checking replication slots...
 slot_name | slot_type | active | restart_lsn 
-----------+-----------+--------+-------------
(0 rows)
[OK]   ✅ Replication slot naming uses underscores (correct)
[INFO] Checking active replication connections...
 application_name | client_addr | state | sync_state 
------------------+-------------+-------+------------
(0 rows)
[INFO] ℹ️  No active replication connections
[INFO] ℹ️  This is a PRIMARY node (no standby.signal)

[INFO] 8️⃣ Checking pg_hba.conf for replication rules...
[WARN] ⚠️ pg_hba.conf not found at /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/pg_hba.conf

[INFO] 9️⃣ Checking Citus configuration...
[INFO] ℹ️  Citus extension not needed on standby (will inherit from primary via replication)

[INFO] 🔟 Checking data directory...
[OK]   ✅ Data directory exists: /data/postgresql/17/user-sau-main-dev/worker-01-standby-01
[OK]   ✅ Data directory size: 4.0K

[INFO] 1️⃣1️⃣ Checking PgBouncer configuration...
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini' as root on web-03.
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/%i/pgbouncer.ini' as root on web-03.
[OK]   ✅ PgBouncer is installed
[INFO]    Version: 1.24.1
2.1.12-stable
c-ares
OpenSSL
yes
[OK]   ✅ PgBouncer service is active: pgbouncer@user-sau-main-dev.service
[WARN] ⚠️ PgBouncer IP service is not active: pgbouncer-ip@user-sau-main-dev.service
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini' as root on web-03.
[WARN] ⚠️ PgBouncer config not found: /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/user-sau-main-dev/userlist.txt' as root on web-03.
[WARN] ⚠️ PgBouncer auth file not found: /etc/pgbouncer/user-sau-main-dev/userlist.txt
[OK]   ✅ PgBouncer is listening on port 6432

[INFO] 1️⃣2️⃣ Enhanced PgBouncer Admin Console Verification...
[INFO] ℹ️  PgBouncer password not found

[INFO] 1️⃣3️⃣ Replicator User Connection Verification...
[INFO] ℹ️  No replication slots configured - skipping replicator verification

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ PostgreSQL verification completed successfully!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Instance:       user-sau-main-dev-worker-01-standby-01
[INFO] Service:        postgresql@user-sau-main-dev-worker-01-standby-01.service
[INFO] Socket:         /var/run/postgresql-user-sau-main-dev-worker-01-standby-01
[INFO] Data Directory: /data/postgresql/17/user-sau-main-dev/worker-01-standby-01
[INFO] Hostname:       db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
[INFO] Port:           5432
[INFO] SSL:            on
[INFO] WAL Level:      logical
[INFO] Citus:          yes
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] ✅ Verification passed for worker-01-standby-01

[INFO] Skipping worker-01-standby-02 - service not configured
[INFO] Skipping worker-01-standby-03 - service not configured
[INFO] Verifying coordinator...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying: coordinator (type: coordinator)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Starting PostgreSQL verification for user-sau-main-dev-coordinator...

[INFO] 1️⃣ Checking systemd service status...
[OK]   ✅ Service postgresql@user-sau-main-dev-coordinator.service is active

[INFO] 2️⃣ Checking PostgreSQL process...
[OK]   ✅ PostgreSQL process is running

[INFO] 3️⃣ Checking socket directory...
[OK]   ✅ Socket directory exists: /var/run/postgresql-user-sau-main-dev-coordinator
total 4
drwxrwsr-x  2 postgres postgres   80 Jan 19 13:49 .
drwxr-xr-x 62 root     root     1620 Jan 19 13:55 ..
srwxrwxrwx  1 postgres postgres    0 Jan 19 13:49 .s.PGSQL.5432
-rw-------  1 postgres postgres  124 Jan 19 13:49 .s.PGSQL.5432.lock

[INFO] 4️⃣ Testing connection via Unix socket...
[OK]   ✅ Socket connection successful
                                                              version                                                              
-----------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.6 (Ubuntu 17.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

[INFO] 5️⃣ Checking SSL certificates...
[2026-01-19 13:55:43 UTC] USER=www-data EUID=0 PID=1419583 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[OK]   ✅ Server certificate exists: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-19 13:55:43 UTC] USER=www-data EUID=0 PID=1419593 ACTION=fsop ARGS=openssl x509 -in /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt -noout -checkend 86400
Certificate will not expire
[OK]   ✅ Server certificate is valid
[2026-01-19 13:55:43 UTC] USER=www-data EUID=0 PID=1419602 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
[OK]   ✅ CA certificate exists: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
[INFO] ℹ️  Client certificates not found at /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[INFO]     (This is OK if using password authentication)

[INFO] 6️⃣ Checking PostgreSQL settings...
[OK]   ✅ SSL is enabled coordinator coordinator
[OK]   ✅ Max connections: 150
[OK]   ✅ Listen addresses: 10.100.1.231
[OK]   ✅ WAL level: logical
[OK]   ✅ Shared preload libraries: shared_preload_libraries

[INFO] 7️⃣ Checking replication configuration...
[INFO] ℹ️  No synchronous standbys configured (single node or async replication)
[INFO] Checking replication slots...
psql: error: connection to server at "db-user-sau-main-dev-postgresql.fastorder.com" (10.100.1.231), port 5432 failed: SSL error: certificate verify failed
connection to server at "db-user-sau-main-dev-postgresql.fastorder.com" (10.100.1.231), port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "10.100.1.231", user "postgres", database "postgres", no encryption
[OK]   ✅ Replication slot naming uses underscores (correct)
[INFO] Checking active replication connections...
  application_name  | client_addr  |   state   | sync_state 
--------------------+--------------+-----------+------------
 Debezium Streaming | 10.100.1.231 | streaming | async
(1 row)
[INFO] ℹ️  Async replication is active
[INFO] ℹ️  This is a PRIMARY node (no standby.signal)

[INFO] 8️⃣ Checking pg_hba.conf for replication rules...
[WARN] ⚠️ pg_hba.conf not found at /data/postgresql/17/user-sau-main-dev/coordinator/pg_hba.conf

[INFO] 9️⃣ Checking Citus configuration...
[OK]   ✅ Citus extension is installed
[OK]   ✅ Citus version: Citus 13.2.0
[OK]   ✅ max_prepared_transactions: 100 (adequate for Citus)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] COORDINATOR-SPECIFIC CHECKS
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking registered workers...
[INFO] ℹ️  Coordinator role verified via pg_dist_node (1 workers registered)
[INFO] Checking coordinator hostname configuration...
[OK]   ✅ Coordinator hostname: -----------------------------------------------------------:----------
[INFO] Checking for stuck prepared transactions...
[OK]   ✅ No stuck Citus prepared transactions
[INFO] Expected workers: 1
[INFO] Registered workers: 1
[OK]   ✅ All 1 worker(s) successfully registered
[INFO] Registered worker nodes:
                         nodename                          | nodeport | groupid | isactive | noderole | shouldhaveshards 
-----------------------------------------------------------+----------+---------+----------+----------+------------------
 db-user-sau-main-dev-postgresql-coordinator.fastorder.com |     5432 |       0 | t        | primary  | f
 db-user-sau-main-dev-postgresql-worker-01.fastorder.com   |     5432 |       1 | t        | primary  | t
(2 rows)

[INFO] Note: groupid=0 is the coordinator, groupid>0 are workers

[INFO] Citus active worker nodes:
                        node_name                        | node_port 
---------------------------------------------------------+-----------
 db-user-sau-main-dev-postgresql-worker-01.fastorder.com |      5432
(1 row)


[INFO] Verifying Citus workers...
[INFO] Checking worker: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
 citus_add_node 
----------------
              2
(1 row)


[INFO] Testing Citus distributed table setup...
[INFO] Checking for blocking locks...
    SELECT pg_terminate_backend(pid)
    FROM pg_stat_activity
    WHERE pid <> pg_backend_pid()
      AND state = 'idle in transaction'
      AND query_start < NOW() - INTERVAL '30 seconds'
      AND datname = current_database();
  
 pg_terminate_backend 
----------------------
(0 rows)

[INFO] Creating demo schema (if needed)...
CREATE SCHEMA
[OK]   ✅ Demo schema ready
[INFO] Creating distributed table 'demo.events'...
CREATE TABLE
[OK]   ✅ Table is already distributed
[INFO] Inserting test data...
INSERT 0 1
[OK]   ✅ Distributed table contains 1 row(s)
[INFO] Checking shard distribution...
[OK]   ✅ Table has 1 shard(s)
[INFO] Shard placement across workers (first 10 shards):
 shardid | nodename | nodeport | shardstate 
---------+----------+----------+------------
(0 rows)
[OK]   ✅ Verified 3 shard placement(s)
[INFO] Testing query routing (EXPLAIN for user_id=42)...
[INFO]    Query plan:         QUERY PLAN        
--------------------------
 Seq Scan on events
   Filter: (user_id = 42)
(2 rows)


[INFO] 🔟 Checking data directory...
[OK]   ✅ Data directory exists: /data/postgresql/17/user-sau-main-dev/coordinator
[OK]   ✅ Data directory size: 4.0K

[INFO] 1️⃣1️⃣ Checking PgBouncer configuration...
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini' as root on web-03.
Failed to print table: Broken pipe
[OK]   ✅ PgBouncer is installed
[INFO]    Version: 1.24.1
2.1.12-stable
c-ares
OpenSSL
yes
[OK]   ✅ PgBouncer service is active: pgbouncer@user-sau-main-dev.service
[WARN] ⚠️ PgBouncer IP service is not active: pgbouncer-ip@user-sau-main-dev.service
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini' as root on web-03.
[WARN] ⚠️ PgBouncer config not found: /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/user-sau-main-dev/userlist.txt' as root on web-03.
[WARN] ⚠️ PgBouncer auth file not found: /etc/pgbouncer/user-sau-main-dev/userlist.txt
[OK]   ✅ PgBouncer is listening on port 6432

[INFO] 1️⃣2️⃣ Enhanced PgBouncer Admin Console Verification...
[INFO] ℹ️  PgBouncer password not found

[INFO] 1️⃣3️⃣ Replicator User Connection Verification...
[INFO] Found 1 replication slot(s) - verifying replicator connectivity...
[WARN] ⚠️ Replicator certificates not found at /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[INFO]    Expected files:
[INFO]    - /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[INFO]    - /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/replicator.crt
[INFO]    - /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/replicator.key
[INFO] Checking pg_hba.conf replicator rules...
[OK]   ✅ Replicator HBA rules found:
 line_number | type | database | user_name | address | auth_method | options | error 
-------------+------+----------+-----------+---------+-------------+---------+-------
(0 rows)
[INFO] Checking active replicator connections in pg_stat_activity...
[WARN] ⚠️ No active replicator connections in pg_stat_activity
[WARN]    This is expected if standbys are not currently connected

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ PostgreSQL verification completed successfully!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Instance:       user-sau-main-dev-coordinator
[INFO] Service:        postgresql@user-sau-main-dev-coordinator.service
[INFO] Socket:         /var/run/postgresql-user-sau-main-dev-coordinator
[INFO] Data Directory: /data/postgresql/17/user-sau-main-dev/coordinator
[INFO] Hostname:       db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[INFO] Port:           5432
[INFO] SSL:            on
[INFO] WAL Level:      logical
[INFO] Citus:          yes
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Failed to print table: Broken pipe

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 💡 OPTIMIZATION OPPORTUNITIES (Optional Enhancements)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1. Enable PgBouncer connection pooling
[INFO]    📦 Benefit: Reduces connection overhead for high-concurrency workloads
[INFO]    ⚡ Use case: When facing connection exhaustion or frequent connect/disconnect cycles
[INFO]    🔧 Action: Enable and configure pgbouncer@user-sau-main-dev.service
[INFO]    📚 Docs: https://www.pgbouncer.org/config.html

[INFO] 2. Enable synchronous replication for zero-data-loss (RPO=0)
[INFO]    🛡️  Benefit: Guaranteed no data loss on primary failure (zero RPO)
[INFO]    ⚖️  Trade-off: Slightly higher write latency (~1-5ms) for durability guarantee
[INFO]    🎯 Use case: Critical data requiring absolute durability across availability zones
[INFO]    🔧 Action: ALTER SYSTEM SET synchronous_standby_names = 'ANY 1 (coordinator_standby_01, coordinator_standby_02)';
[INFO]    ⚠️  Note: Requires at least one standby to be available for writes to commit
[INFO]    📚 Docs: https://www.postgresql.org/docs/current/warm-standby.html#SYNCHRONOUS-REPLICATION

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ℹ️  These are optional enhancements for production-scale deployments
[INFO] ℹ️  Current configuration is fully functional and ready for production
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] ✅ Verification passed for coordinator


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📊 PRODUCTION READINESS CHECKS (Step 04 & 05)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔍 Checking Monitoring Setup (postgres_exporter or observability cell)...

[INFO] ℹ️  Monitoring can be configured via:
[INFO]    • Local postgres_exporter (step 04-monitoring-setup.sh)
[INFO]    • Observability Cell integration (step 02-observability-cell)

[ OK ] ✅ postgres_exporter is installed
[INFO]    Version: 0.10.1-1ubuntu0.22.04.3
[ OK ] ✅ postgres_exporter-user-sau-main-dev.service is running
[WARN] ⚠️  Metrics endpoint not responding
[INFO] ℹ️  Monitoring user 'postgres_exporter' not found in PostgreSQL
[INFO]    This is expected if using observability cell remote monitoring
[INFO] ℹ️  Monitoring check passed (local or observability cell)

[INFO] 🔍 Checking Backup Setup (pgBackRest + WAL archiving)...

[ OK ] ✅ pgBackRest is installed
[INFO]    Version: pgBackRest 2.56.0
[ OK ] ✅ WAL archiving is enabled (archive_mode=on)
[ OK ] ✅ archive_command is configured for pgBackRest
[INFO]    Command: timeout 30 /usr/bin/pgbackrest --stanza=user-sau-main-dev-coordinator archive-push %p
[ OK ] ✅ pgBackRest configuration exists
[ OK ] ✅ pgBackRest stanza 'user-sau-main-dev-coordinator' is initialized
[ OK ] ✅ Backups exist (4 full backup(s))
[INFO]    Latest backup info:
                 timestamp start/stop: 2026-01-19 13:49:06+00 / 2026-01-19 13:49:14+00
                 wal start/stop: 000000010000000000000003 / 000000010000000000000003
                 database size: 37.5MB, database backup size: 37.5MB
                 repo1: backup set size: 5.6MB, backup size: 5.6MB
     
             full backup: 20260119-134925F
                 timestamp start/stop: 2026-01-19 13:49:25+00 / 2026-01-19 13:49:32+00
                 wal start/stop: 000000010000000000000006 / 000000010000000000000006
                 database size: 37.5MB, database backup size: 37.5MB
                 repo1: backup set size: 5.6MB, backup size: 5.6MB
[ OK ] ✅ Automated backup cron jobs are configured
[INFO]    Schedule:
     0 2 * * 0 root /usr/local/bin/pgbackrest-full-backup-user-sau-main-dev.sh
     0 2 * * 1-6 root /usr/local/bin/pgbackrest-diff-backup-user-sau-main-dev.sh
[ OK ] ✅ Backup directory exists: /var/lib/pgbackrest
[INFO]    Total backup size: 2.6G

[INFO] 🔍 Checking Worker Backup Coverage...

[INFO] ℹ️  Worker backups are optional for development environments
[INFO]    For production, ensure all workers have backup coverage

[INFO] Checking worker 1/1: worker-01...
[WARN] ⚠️  Worker worker-01 stanza exists but status unknown
[INFO] ℹ️  Incomplete worker backup coverage (0/1) - OK for dev

[INFO] 🔍 Checking Synchronous Replication (RPO=0)...

[INFO] ℹ️  Synchronous replication (RPO=0) is optional for development
[INFO]    For production with zero data loss requirement, enable sync replication

[INFO] ℹ️  Worker worker-01 synchronous replication NOT configured
[INFO]    └─ synchronous_commit: on
[INFO]    └─ synchronous_standby_names: 
[ OK ] ✅ All workers have synchronous replication (RPO=0)

[INFO] 🔍 Checking Connection and Memory Optimization...

[ OK ] ✅ Coordinator max_connections optimized: 150
[ OK ] ✅ Coordinator work_mem optimized: 8MB
[ OK ] ✅ Worker worker-01 max_connections optimized: 100
[ OK ] ✅ Worker worker-01 work_mem optimized: 8MB
[ OK ] ✅ All instances have optimized connection and memory settings

[INFO] 🔍 Checking Optimizations...

[ OK ] ✅ Citus coordinator host configured: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[ OK ] ✅ Periodic integrity checks configured
[INFO]    └─ Daily checks: 3, Weekly verify: 3
[WARN] ⚠️  Backup schedule NOT staggered (all at :00)
[INFO]    Optimize with: ./setup/04-postgresql/steps/04-production-optimizations.sh
[2026-01-19 13:56:00 UTC] USER=www-data EUID=0 PID=1420818 ACTION=fsop ARGS=test -f /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-19 13:56:00 UTC] USER=www-data EUID=0 PID=1420828 ACTION=fsop ARGS=grep -q ## Cipher Key Management /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[WARN] ⚠️  Cipher key management documentation missing
[INFO]    Add with: ./setup/04-postgresql/steps/04-production-optimizations.sh
[INFO] ℹ️  Offsite backup (repo2) not configured (optional for production)
[INFO]    Setup guide: ./setup/05-db/engine/postgresql/steps/14-setup-offsite-backup.sh
[WARN] ⚠️  Some production optimizations incomplete

[INFO] 🔍 Checking Citus Maintenance Daemon Health...

[INFO] Checking for stuck Citus Maintenance Daemons...
[ OK ] ✅ Citus Maintenance Daemons are healthy
[INFO] Checking for stuck distributed table operations...
[ OK ] ✅ No stuck distributed table operations
[INFO] Testing distributed table operations (10s timeout)...
[WARN] ⚠️  CRITICAL: Distributed table test TIMED OUT (10s)
[WARN]    Citus cluster is NOT operational - distributed tables cannot be created
[WARN]    This confirms maintenance daemons are stuck
[WARN]    
[WARN]    🔧 ACTION REQUIRED: Restart coordinator before using Citus
[WARN]       sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-coordinator.service
[INFO] Checking for lock contention...
[ OK ] ✅ No lock contention detected
[INFO] Checking for lingering prepared transactions...
[ OK ] ✅ No lingering prepared transactions

[WARN] ⚠️  Citus cluster has health issues - see warnings above
[WARN]    
[WARN]    ⚡ IMMEDIATE ACTION: Restart coordinator to restore Citus functionality
[WARN]       sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev.service

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📋 PRODUCTION READINESS SUMMARY
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Component                 Status          Production Ready?   
───────────────────────── ─────────────── ────────────────────
Citus Cluster             ✅ Operational YES                 
High Availability         ✅ Configured  YES                 
SSL/TLS Security          ✅ Enabled     YES                 
PgBouncer                 ✅ Running     YES                 
Monitoring                ✅ Operational YES                 
Backups (Coordinator)     ✅ Configured  YES                 
Backups (Workers)         ✅ Configured  YES                 
Sync Replication (RPO=0)  ✅ Enabled     YES                 
Connection Optimization   ✅ Configured  YES                 
Optimizations             ⚠️  Incomplete OPTIONAL            

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[ OK ] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] 🎉 PRODUCTION READY: 100% (3/3 critical checks passed)
[ OK ] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ All critical components are operational and production-ready

[INFO] Next steps:
[INFO]   1. Configure Prometheus to scrape metrics: http://localhost:9248/metrics
[INFO]   2. Import Grafana dashboards for PostgreSQL + Citus monitoring
[INFO]   3. Setup alerting rules for critical metrics
[INFO]   4. Schedule regular restore drills (monthly)
[INFO]   5. Review /var/www/html/skeleton.dev.fastorder.com/fixing/scripts/PRODUCTION_READINESS.md

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✓ Verification process completed successfully
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 18-production-optimizations.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PostgreSQL Production Optimizations
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Environment: user-sau-main-dev
[INFO] Enable Sync Replication: --auto

[INFO] 1️⃣ Configuring Citus coordinator hostname...
[ OK ] ✅ Coordinator hostname already configured: db-user-sau-main-dev-postgresql-coordinator.fastorder.com

[INFO] 2️⃣ Configuring synchronous replication for RPO=0...
[INFO] Synchronous replication NOT enabled (use './04-production-optimizations.sh yes' to enable)
[INFO] Current configuration: async replication (RPO > 0)
[INFO] 
[INFO] To enable safely after deployment:
[INFO]   /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/enable_sync_replication_safe.sh \
[INFO]     /var/run/postgresql-user-sau-main-dev-worker-01 worker_01_standby_01

[INFO] 3️⃣ Adding periodic integrity check cron jobs...
[2026-01-19 13:56:03 UTC] USER=www-data EUID=0 PID=1421099 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-integrity-user-sau-main-dev
[ OK ] ✅ Integrity check cron jobs configured
[INFO]    Daily checks: 02:15, 03:30, 04:45 (coordinator, worker-01, worker-02)
[INFO]    Weekly verify: Sundays at same times

[INFO] 4️⃣ Updating backup schedule with staggered timing...
[ OK ] ✅ Backup schedule staggered:
[INFO]    Coordinator: 02:05 (full: Sun, diff: Mon-Sat)
[INFO]    Worker-01:   03:10 (full: Sun, diff: Mon-Sat)
[INFO]    Worker-02:   04:15 (full: Sun, diff: Mon-Sat)

[INFO] 5️⃣ Documenting cipher key backup procedures...
[2026-01-19 13:56:03 UTC] USER=www-data EUID=0 PID=1421134 ACTION=fsop ARGS=test -f /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
Sorry, user www-data is not allowed to execute '/usr/bin/grep -q ## Cipher Key Management /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md' as root on web-03.
[ OK ] ✅ Cipher key documentation added to /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md

[INFO] 6️⃣ Checking offsite backup configuration...
[INFO] ℹ️  Offsite backup (repo2) is NOT configured
[INFO]    Configuration example: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[INFO]    Setup instructions: ./setup/04-postgresql/steps/08-setup-offsite-backup.sh
[ OK ] ✅ Offsite backup example available: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Production Optimizations Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[ OK ] Applied optimizations:
[ OK ]   1. ✅ Citus coordinator hostname
[ OK ]   2. ⏭️  Synchronous replication (RPO=0)
[ OK ]   3. ✅ Periodic integrity checks (daily + weekly)
[ OK ]   4. ✅ Staggered backup schedule (reduced load spikes)
[ OK ]   5. ✅ Cipher key backup documentation
[ OK ]   6. ✅ Offsite backup (repo2) example configuration

[INFO] Next steps:
[INFO]   1. Backup cipher keys to secure vault immediately
[INFO]   2. Set up S3/MinIO for offsite backups:
[INFO]      - Instructions: ./setup/04-postgresql/steps/08-setup-offsite-backup.sh
[INFO]      - Example config: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[INFO]   3. Configure alerting for backup failures (cron exit codes)
[INFO]   4. Test restore drill from offsite repository
[INFO]   5. Enable RPO=0 if needed: ./04-production-optimizations.sh yes

[ OK ] System is now production-grade! 🎉

✓ ✔ PostgreSQL creation completed
[INFO] Cleaning up temporary files...
[INFO] Starting cleanup of temporary files...
[INFO] Cleaning up SSL temp files for user-sau-main-dev...
[INFO] Cleaning up old provisioning logs...
[INFO] Cleaning up old configuration backups...
✓ ✔ Cleanup completed

✓ ✅ Database infrastructure (postgresql) setup completed successfully

[INFO] Using environment from web interface: user-sau-main-dev
[2026-01-19 13:56:04] Using web-provided environment: user-sau-main-dev
[2026-01-19 13:56:04] Service: user, Zone: sau, Branch: main, Env: dev
✓ Environment initialized successfully (mode: general)
[INFO] Starting finalizing setup process...
[INFO] Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps
[INFO] Environment: user-sau-main-dev

[INFO] Found 3 step(s) to execute

[INFO] 📦 Step 1/3: enable_disable_all_applications...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
════════════════════════════════════════════════════════════════════════════════
  Environment Services Management
════════════════════════════════════════════════════════════════════════════════
  Environment:  user-sau-main-dev
  Action:       enable
  Triggered by: false
════════════════════════════════════════════════════════════════════════════════

🔍 Scanning for environment-specific services...
✅ Found 7 services for environment: user-sau-main-dev

📋 Services to enable:
────────────────────────────────────────────────────────────────────────────────
  • confluent-connect-user-sau-main-dev_coordinator.service      [active/unmasked/enabled]
  • confluent-kraft-user-sau-main-dev_coordinator.service        [active/unmasked/enabled]
  • elasticsearch@user-sau-main-dev-node-01.service              [active/unmasked/enabled]
  • pgbouncer@user-sau-main-dev.service                          [active/unmasked/enabled]
  • postgresql@user-sau-main-dev-coordinator.service             [active/unmasked/enabled]
  • postgresql@user-sau-main-dev-worker-01-standby-01.service    [active/unmasked/enabled]
  • postgresql@user-sau-main-dev-worker-01.service               [active/unmasked/enabled]
────────────────────────────────────────────────────────────────────────────────


❌ Cancelled by user
[OK] ✅ Step 1 completed: 01-enable_disable_all_applications.sh

[INFO] 📦 Step 2/3: verify monitoring...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Monitoring Verification for user-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Detecting installed services...
Failed to print table: Broken pipe
[OK] ✓ PostgreSQL detected
Failed to print table: Broken pipe
[OK] ✓ Elasticsearch detected
Failed to print table: Broken pipe
[OK] ✓ Kafka detected
Failed to print table: Broken pipe
[OK] ✓ PgBouncer detected

[INFO] Services to verify: postgresql elasticsearch kafka pgbouncer

[INFO] 2️⃣ Verifying exporters are running...
[OK] ✓ PostgreSQL exporter is running
[OK] ✓ Elasticsearch exporter is running
[OK] ✓ Kafka JMX exporter is running
[OK] ✓ PgBouncer exporter is running

[INFO] 3️⃣ Verifying Prometheus configuration...
[2026-01-19 13:56:06 UTC] USER=www-data EUID=0 PID=1421622 ACTION=passthru ARGS=grep -q job_name: 'postgresql' /etc/prometheus/obs-user-sau-main-dev/prometheus.yml
[OK] ✓ postgresql is configured in Prometheus
[OK] ✓ elasticsearch is configured in Prometheus
[2026-01-19 13:56:06 UTC] USER=www-data EUID=0 PID=1421666 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-user-sau-main-dev/prometheus.yml
[WARN] ⚠️  kafka is not configured in Prometheus scrape targets
[2026-01-19 13:56:07 UTC] USER=www-data EUID=0 PID=1421689 ACTION=passthru ARGS=grep -q job_name: 'pgbouncer' /etc/prometheus/obs-user-sau-main-dev/prometheus.yml
[WARN] ⚠️  pgbouncer is not configured in Prometheus scrape targets

[INFO] 4️⃣ Verifying Prometheus is actively scraping...
[OK] ✓ Prometheus is running
[OK] ✓ postgresql target is UP in Prometheus
[OK] ✓ elasticsearch target is UP in Prometheus
[WARN] ⚠️  kafka target is not UP in Prometheus (may still be initializing)
[WARN] ⚠️  pgbouncer target is not UP in Prometheus (may still be initializing)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Monitoring Verification Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[WARN] Some monitoring issues were detected:

[WARN] Prometheus Configuration Issues:
  - kafka not configured in Prometheus
  - pgbouncer not configured in Prometheus

[WARN] Automatically running monitoring setup scripts to fix issues...

[INFO] Running Kafka monitoring setup...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Kafka Monitoring Integration for user-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-user-sau-main-dev
[OK]   Observability cell endpoints registered for user-sau-main-dev
[OK]   ✓ Observability cell is ready

[INFO] 2️⃣ Setting up Kafka JMX exporter integration...
[INFO] JMX Exporter port calculated for user-sau-main-dev: 9357 (offset: 49)
[INFO] Checking observability cell readiness: obs-user-sau-main-dev
[OK]   Observability cell endpoints registered for user-sau-main-dev
[INFO] Setting up Kafka JMX exporter for user-sau-main-dev
[INFO] JMX Prometheus Java Agent already exists at /opt/kafka/libs/jmx_prometheus_javaagent.jar
[2026-01-19 13:56:12 UTC] USER=www-data EUID=0 PID=1421886 ACTION=passthru ARGS=mv /tmp/jmx_exporter.yml /opt/kafka/config/jmx_exporter.yml
[2026-01-19 13:56:12 UTC] USER=www-data EUID=0 PID=1421897 ACTION=passthru ARGS=chmod 644 /opt/kafka/config/jmx_exporter.yml
[OK]   JMX exporter configuration created at /opt/kafka/config/jmx_exporter.yml
[OK]   JMX exporter configuration created
[INFO] Configuring Kafka systemd services to use JMX exporter...
[2026-01-19 13:56:12 UTC] USER=www-data EUID=0 PID=1421921 ACTION=fsop ARGS=test -f /etc/systemd/system/[2026-01-19
[INFO] All Kafka services already configured with JMX exporter
[OK]   Kafka JMX exporter integration complete
[INFO] Metrics endpoint: http://142.93.238.16:9357/metrics
[INFO] Prometheus will automatically scrape: https://metrics-user-sau-main-dev.fastorder.com:9090
[INFO] View dashboards at: https://dashboards-user-sau-main-dev.fastorder.com
[OK]   ✓ Kafka JMX exporter integration complete
[INFO] Configuring KAFKA_OPTS environment variable for kafka user...
[2026-01-19 13:56:12 UTC] USER=www-data EUID=0 PID=1421942 ACTION=passthru ARGS=grep -q KAFKA_OPTS.*javaagent.*jmx_prometheus_javaagent.*=9357: /home/kafka/.bashrc
[2026-01-19 13:56:13 UTC] USER=www-data EUID=0 PID=1421974 ACTION=passthru ARGS=sed -i /export KAFKA_OPTS=.*jmx_prometheus_javaagent/d /home/kafka/.bashrc
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[OK]   ✓ KAFKA_OPTS configured in /home/kafka/.bashrc (port 9357)
[INFO] 2.5️⃣ Enabling JMX exporter in Kafka systemd service...
[2026-01-19 13:56:13 UTC] USER=www-data EUID=0 PID=1422008 ACTION=passthru ARGS=grep -q javaagent.*jmx_prometheus_javaagent /etc/systemd/system/confluent-kraft-user-sau-main-dev_coordinator.service
[OK]   ✓ JMX exporter already enabled in Kafka systemd services
[INFO] 2.6️⃣ Configuring Prometheus to scrape Kafka metrics...
[2026-01-19 13:56:13 UTC] USER=www-data EUID=0 PID=1422031 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-user-sau-main-dev/prometheus.yml
[INFO] Adding Kafka scrape target to Prometheus configuration...
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[2026-01-19 13:56:13 UTC] USER=www-data EUID=0 PID=1422064 ACTION=passthru ARGS=sed -i /# Prometheus self-monitoring/r /tmp/prometheus_kafka_add.yml /etc/prometheus/obs-user-sau-main-dev/prometheus.yml
[ERROR] Invalid Prometheus configuration - rolling back
[2026-01-19 13:56:13 UTC] USER=www-data EUID=0 PID=1422105 ACTION=passthru ARGS=sed -i /job_name: 'kafka'/,+6d /etc/prometheus/obs-user-sau-main-dev/prometheus.yml

[INFO] 3️⃣ Registering Kafka nodes to monitoring database...
[INFO] Detected Kafka version: 3.9.1
[INFO] Registering Kafka Broker to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Kafka Broker
[INFO]   Identifier:        user-sau-main-dev-broker-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                142.93.238.16
[INFO]   Port:              9092
[INFO]   FQDN:              eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: f13110f1-e44a-4c54-a00b-2cf550959aef
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[OK]   ✓ Kafka broker registered
[INFO] Registering Kafka Connect to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Kafka Connect
[INFO]   Identifier:        user-sau-main-dev-connect-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8083
[INFO]   FQDN:              eventbus-user-sau-main-dev-kafka-connect.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 44032700-39d0-4bd0-aa84-ed929b0f2345
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[OK]   ✓ Kafka Connect registered
[INFO] Schema Registry not running, skipping registration

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Kafka Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Metrics: http://localhost:9357/metrics
[INFO] Prometheus: https://metrics-user-sau-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-user-sau-main-dev.fastorder.com
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✓ Kafka monitoring setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Step 2 completed: 02-verify-monitoring.sh

[INFO] 📦 Step 3/3: register backup infrastructure...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔧 Registering Core Services & Backup Infrastructure for user-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Registering Main App...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Main App
[INFO]   Identifier:        user-sau-main-dev-main-app
[INFO]   Identifier Parent: application
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8080
[INFO]   FQDN:              app-user-sau-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: f1a71872-2c8a-4fe5-b8ee-dabaaf8bc0c0
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
/opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps/03-register-backup-infrastructure.sh: line 70: ok: command not found

[INFO] 2️⃣ Registering Audit Service...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Audit Service
[INFO]   Identifier:        user-sau-main-dev-audit
[INFO]   Identifier Parent: application
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8081
[INFO]   FQDN:              audit-user-sau-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: d6a172a9-01cd-4d3f-a7b7-642b3294a19e
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
/opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps/03-register-backup-infrastructure.sh: line 85: ok: command not found

[INFO] 3️⃣ Registering PostgreSQL Backup Node...
[ERROR] Invalid identifier format: backup-db
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register PostgreSQL backup node (non-blocking)

[INFO] 4️⃣ Registering Elasticsearch Backup Node...
[ERROR] Invalid identifier format: backup-search
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Elasticsearch backup node (non-blocking)

[INFO] 5️⃣ Registering Kafka Backup Node...
[ERROR] Invalid identifier format: backup-eventbus
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Kafka backup node (non-blocking)

[INFO] 6️⃣ Registering Backup Orchestrator...
[ERROR] Invalid identifier format: backup-orchestrator
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Backup orchestrator (non-blocking)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Core Services & Backup Infrastructure Registration Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Registered core services:
[INFO]   🚀 main-app            → Core application service
[INFO]   📋 audit               → Centralized audit logging (WORM)

[INFO] Registered backup nodes:
[INFO]   📦 backup-db           → PostgreSQL backup (pgBackRest, PITR)
[INFO]   📦 backup-search       → Elasticsearch snapshots (ILM, S3)
[INFO]   📦 backup-eventbus     → Kafka log segments (replication)
[INFO]   📦 backup-orchestrator → Central backup coordination

[INFO] Dashboard: https://skeleton.dev.fastorder.com/dashboard/monitoring
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Step 3 completed: 03-register-backup-infrastructure.sh


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ finalizing setup completed successfully!
[OK] Executed all 3 steps
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Environment: user-sau-main-dev
[INFO] Service: user
[INFO] Zone: sau
[INFO] Branch: main
[INFO] Env: dev