📊 Provisioning Job Status

[INFO] Using database engine from DB_ENGINE environment variable: postgresql
[INFO] Cleaning up any existing locks...

Starting database engine: postgresql
═══════════════════════════════════════════════

[INFO] Loaded from topology.json: identity-sau-main-dev
[2026-01-02 11:36:17] Loaded environment: identity-sau-main-dev
[2026-01-02 11:36:17] Service: identity, Zone: sau, Branch: main, Env: dev
[2026-01-02 11:36:17] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-02 11:36:17] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-02 11:36:17] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[INFO] Observability cell verified for identity-sau-main-dev
[INFO] Monitoring will be configured after PostgreSQL deployment (step 10-monitoring-setup.sh)
[INFO] Citus mode ENABLED
[INFO] → Coordinator + 1 worker(s) + 1 standby node(s) per worker
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up coordinator (Citus control plane)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-02 11:36:18 UTC] USER=unknown EUID=33 PID=2015868 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 11:36:18 UTC] USER=unknown EUID=33 PID=2015875 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 11:36:18 UTC] USER=unknown EUID=33 PID=2015882 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-02 11:36:18 UTC] USER=unknown EUID=33 PID=2015889 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 11:36:18 UTC] USER=unknown EUID=33 PID=2015896 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-02 11:36:18 UTC] USER=unknown EUID=33 PID=2015903 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38 JOB_UUID=25733ea8-d48a-48b5-970c-0036cccf455b

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] PostgreSQL IP: 10.100.1.213
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

[INFO] Adding /etc/hosts entries for coordinator...
[INFO]   1. db-identity-sau-main-dev-postgresql.fastorder.com → 10.100.1.213 (primary/short)
[INFO]   2. db-identity-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.213 (compatibility)

[INFO]   ✅ db-identity-sau-main-dev-postgresql.fastorder.com already exists with correct IP
[INFO]   ✅ db-identity-sau-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.213    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  10.100.1.213    db-identity-sau-main-dev-postgresql.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 11:36:21 UTC] USER=www-data EUID=0 PID=2016016 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator and /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:36:21 UTC] USER=www-data EUID=0 PID=2016025 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-02 11:36:22 UTC] USER=www-data EUID=0 PID=2016035 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-2015983
[2026-01-02 11:36:22 UTC] USER=www-data EUID=0 PID=2016044 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-2015983/ra_root.crt
[2026-01-02 11:36:22 UTC] USER=www-data EUID=0 PID=2016053 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-2015983/ra_root.key
[2026-01-02 11:36:22 UTC] USER=www-data EUID=0 PID=2016064 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-2015983/ra_root.crt
[2026-01-02 11:36:22 UTC] USER=www-data EUID=0 PID=2016074 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-2015983/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-02 11:36:23 UTC] USER=www-data EUID=0 PID=2016125 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2015983/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 11:36:23 UTC] USER=www-data EUID=0 PID=2016134 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2015983/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 11:36:23 UTC] USER=www-data EUID=0 PID=2016143 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-02 11:36:23 UTC] USER=www-data EUID=0 PID=2016152 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2015983/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:23 UTC] USER=www-data EUID=0 PID=2016161 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:23 UTC] USER=www-data EUID=0 PID=2016170 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:23 UTC] USER=www-data EUID=0 PID=2016179 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 11:36:23 UTC] USER=www-data EUID=0 PID=2016190 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 11:36:24 UTC] USER=www-data EUID=0 PID=2016199 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 11:36:24 UTC] USER=www-data EUID=0 PID=2016208 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 11:36:24 UTC] USER=www-data EUID=0 PID=2016217 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 11:36:24 UTC] USER=www-data EUID=0 PID=2016226 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:36:24 UTC] USER=www-data EUID=0 PID=2016235 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        coordinator
Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:36:24 UTC] USER=www-data EUID=0 PID=2016289 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-01-02 11:36:24 UTC] USER=www-data EUID=0 PID=2016298 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 11:36:24 UTC] USER=www-data EUID=0 PID=2016307 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-01-02 11:36:24 UTC] USER=www-data EUID=0 PID=2016316 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 11:36:24 UTC] USER=www-data EUID=0 PID=2016325 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016340 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016350 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016359 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016368 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016377 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016386 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016395 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016404 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016413 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016422 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016431 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016440 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016449 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016458 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016469 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016478 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016505 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016514 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016523 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016532 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016541 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:25 UTC] USER=www-data EUID=0 PID=2016550 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016559 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016568 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016577 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016586 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016596 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016606 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016616 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016625 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016634 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016643 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016652 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016661 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016670 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016679 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016688 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016697 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016707 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016717 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016727 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016736 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016745 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016754 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016763 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016772 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016781 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016790 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016799 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:36:26 UTC] USER=www-data EUID=0 PID=2016808 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016817 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016827 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016837 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016846 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016857 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016866 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016875 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016884 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016911 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016920 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016929 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016939 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016949 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016958 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016970 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016980 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016989 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2016998 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2017007 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2017016 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:36:27 UTC] USER=www-data EUID=0 PID=2017027 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:36:28 UTC] USER=www-data EUID=0 PID=2017070 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-01-02 11:36:28 UTC] USER=www-data EUID=0 PID=2017079 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 11:36:28 UTC] USER=www-data EUID=0 PID=2017088 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-01-02 11:36:28 UTC] USER=www-data EUID=0 PID=2017097 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 11:36:28 UTC] USER=www-data EUID=0 PID=2017106 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017120 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017132 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017141 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017150 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017168 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017177 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017186 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017195 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017204 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017213 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017222 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017231 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017240 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017252 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017261 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017270 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017279 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017305 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017314 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017323 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017332 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017341 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017350 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 11:36:29 UTC] USER=www-data EUID=0 PID=2017359 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017368 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017377 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017386 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017395 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017405 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017415 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017424 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017433 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017442 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017451 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017460 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017469 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017478 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017487 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017496 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017505 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017515 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017525 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017534 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017543 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017552 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017561 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017570 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017579 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017588 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017597 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:36:30 UTC] USER=www-data EUID=0 PID=2017606 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017615 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017625 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017635 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017644 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017653 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017662 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017671 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017680 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017689 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017698 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017707 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017716 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017737 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017747 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017756 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017765 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017774 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017783 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017792 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017801 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017810 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:36:31 UTC] USER=www-data EUID=0 PID=2017819 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-coordinator-postgresql environment: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com (10.100.1.213)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.213
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] Data dir:   /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[INFO] Port:       5432
[INFO] Hostname:   db-identity-sau-main-dev-postgresql-coordinator
[2026-01-02 11:36:33 UTC] USER=www-data EUID=0 PID=2017926 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:36:33 UTC] USER=www-data EUID=0 PID=2017948 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:36:33 UTC] USER=www-data EUID=0 PID=2017969 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:36:34 UTC] USER=www-data EUID=0 PID=2017990 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 11:36:34 UTC] USER=www-data EUID=0 PID=2018034 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator and /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:36:34 UTC] USER=www-data EUID=0 PID=2018043 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-02 11:36:34 UTC] USER=www-data EUID=0 PID=2018053 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-2017997
[2026-01-02 11:36:34 UTC] USER=www-data EUID=0 PID=2018062 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-2017997/ra_root.crt
[2026-01-02 11:36:34 UTC] USER=www-data EUID=0 PID=2018071 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-2017997/ra_root.key
[2026-01-02 11:36:34 UTC] USER=www-data EUID=0 PID=2018080 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-2017997/ra_root.crt
[2026-01-02 11:36:34 UTC] USER=www-data EUID=0 PID=2018089 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-2017997/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-02 11:36:35 UTC] USER=www-data EUID=0 PID=2018125 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2017997/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 11:36:35 UTC] USER=www-data EUID=0 PID=2018134 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2017997/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 11:36:35 UTC] USER=www-data EUID=0 PID=2018143 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-02 11:36:35 UTC] USER=www-data EUID=0 PID=2018152 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2017997/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:35 UTC] USER=www-data EUID=0 PID=2018161 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:36 UTC] USER=www-data EUID=0 PID=2018170 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:36:36 UTC] USER=www-data EUID=0 PID=2018179 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 11:36:36 UTC] USER=www-data EUID=0 PID=2018190 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 11:36:36 UTC] USER=www-data EUID=0 PID=2018201 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 11:36:36 UTC] USER=www-data EUID=0 PID=2018210 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 11:36:36 UTC] USER=www-data EUID=0 PID=2018219 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 11:36:36 UTC] USER=www-data EUID=0 PID=2018237 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        coordinator
Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-02 11:36:36 UTC] USER=www-data EUID=0 PID=2018266 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 11:36:36 UTC] USER=www-data EUID=0 PID=2018275 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 11:36:36 UTC] USER=www-data EUID=0 PID=2018284 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-02 11:36:36 UTC] USER=www-data EUID=0 PID=2018305 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 11:36:36 UTC] USER=www-data EUID=0 PID=2018331 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-02 11:36:37 UTC] USER=www-data EUID=0 PID=2018362 ACTION=fsop ARGS=rm -rf /var/run/postgresql-identity-sau-main-dev-coordinator
[OK]   No conflicting Postgres left on port 5432
[OK]   Using postgres password from vault provider
[2026-01-02 11:36:40 UTC] USER=www-data EUID=0 PID=2018452 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.QigRhD
[2026-01-02 11:36:40 UTC] USER=www-data EUID=0 PID=2018474 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 11:36:40 UTC] USER=www-data EUID=0 PID=2018497 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 11:36:40 UTC] USER=www-data EUID=0 PID=2018521 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/identity-sau-main-dev
[INFO] Initializing cluster in /var/lib/postgresql/17/identity-sau-main-dev/coordinator (SCRAM; pwfile)
[WARN] Removing existing data directory: /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 11:36:40 UTC] USER=www-data EUID=0 PID=2018542 ACTION=fsop ARGS=rm -rf /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 11:36:41 UTC] USER=www-data EUID=0 PID=2018565 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 11:36:41 UTC] USER=www-data EUID=0 PID=2018586 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 11:36:41 UTC] USER=www-data EUID=0 PID=2018607 ACTION=fsop ARGS=chmod 700 /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 11:36:41 UTC] USER=www-data EUID=0 PID=2018628 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-02 11:36:41 UTC] USER=www-data EUID=0 PID=2018649 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-02 11:36:41 UTC] USER=www-data EUID=0 PID=2018670 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-02 11:36:41 UTC] USER=www-data EUID=0 PID=2018679 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /var/lib/postgresql/17/identity-sau-main-dev/coordinator --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.QigRhD
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/17/identity-sau-main-dev/coordinator ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /var/lib/postgresql/17/identity-sau-main-dev/coordinator -l logfile start

[OK]   initdb complete
[2026-01-02 11:36:42 UTC] USER=www-data EUID=0 PID=2018722 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.QigRhD
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-02 11:36:43 UTC] USER=www-data EUID=0 PID=2018770 ACTION=fsop ARGS=cp /tmp/tmp.S1iHvGiALC /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
[2026-01-02 11:36:43 UTC] USER=www-data EUID=0 PID=2018791 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
[2026-01-02 11:36:43 UTC] USER=www-data EUID=0 PID=2018812 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 11:36:43 UTC] USER=www-data EUID=0 PID=2018837 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.R8hoA2 /etc/systemd/system/postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 11:36:43 UTC] USER=www-data EUID=0 PID=2018859 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@identity-sau-main-dev-coordinator.service
[OK]   systemd unit written
[2026-01-02 11:36:43 UTC] USER=www-data EUID=0 PID=2018880 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 11:36:43 UTC] USER=www-data EUID=0 PID=2018901 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 11:36:43 UTC] USER=www-data EUID=0 PID=2018922 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-02 11:36:45 UTC] USER=www-data EUID=0 PID=2019047 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-02 11:36:46 UTC] USER=www-data EUID=0 PID=2019087 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Provisioning application database and Debezium role (if not exists)...
[INFO] Checking if database fastorder_identity_sau_main_dev_db exists...
[INFO] DB check result: exit_code=0, output='[2026-01-02 11:36:47 UTC] USER=www-data EUID=0 PID=2019254 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_identity_sau_main_dev_db''
[INFO] Creating database fastorder_identity_sau_main_dev_db...
[2026-01-02 11:36:48 UTC] USER=www-data EUID=0 PID=2019278 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_identity_sau_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[OK]   Database fastorder_identity_sau_main_dev_db created
[INFO] Checking if role debezium_user exists...
[INFO] Role check result: exit_code=0, output='[2026-01-02 11:36:48 UTC] USER=www-data EUID=0 PID=2019306 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[INFO] Creating role debezium_user...
[2026-01-02 11:36:48 UTC] USER=www-data EUID=0 PID=2019333 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'omIuI1zK+GZwBSKfFJllBw4P';
CREATE ROLE
[OK]   Role debezium_user created
[2026-01-02 11:36:49 UTC] USER=www-data EUID=0 PID=2019358 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_identity_sau_main_dev_db" TO debezium_user;
GRANT
[OK]   Application DB (fastorder_identity_sau_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[INFO] Applying connection and memory optimizations...
[INFO] Current settings: max_connections=100, work_mem=4MB
[INFO] Target settings (coordinator): max_connections=150, work_mem=8MB
[2026-01-02 11:36:49 UTC] USER=www-data EUID=0 PID=2019439 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 150;
ALTER SYSTEM
[2026-01-02 11:36:49 UTC] USER=www-data EUID=0 PID=2019462 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-02 11:36:50 UTC] USER=www-data EUID=0 PID=2019485 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   Settings applied to postgresql.auto.conf
[2026-01-02 11:36:50 UTC] USER=www-data EUID=0 PID=2019500 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/standby.signal
[INFO] Service recently started (4s ago) - restarting to apply max_connections...
[INFO] Stopping service...
[2026-01-02 11:36:50 UTC] USER=www-data EUID=0 PID=2019522 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[INFO] Waiting for port 5432 to be released...
[OK]   Port 5432 released
[INFO] Starting service...
[2026-01-02 11:36:53 UTC] USER=www-data EUID=0 PID=2019895 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 11:36:59 UTC] USER=www-data EUID=0 PID=2019969 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[OK]   ✅ Optimization complete: max_connections=150, work_mem=8MB
[INFO] Setting postgres password via centralized script... for coordinator
[INFO] Temporarily disabling synchronous_commit on coordinator for password setting...
[OK]   Disabled synchronous_commit (was: on)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Initial setup: Using password from initdb
✓ PostgreSQL password already set during initdb
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/coordinator
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
[INFO] Restoring synchronous_commit on coordinator...
[OK]   Restored synchronous_commit to: on
[OK]   Password set and persisted
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] PostgreSQL IP: 10.100.1.213
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

[INFO] Adding /etc/hosts entries for coordinator...
[INFO]   1. db-identity-sau-main-dev-postgresql.fastorder.com → 10.100.1.213 (primary/short)
[INFO]   2. db-identity-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.213 (compatibility)

[INFO]   ✅ db-identity-sau-main-dev-postgresql.fastorder.com already exists with correct IP
[INFO]   ✅ db-identity-sau-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.213    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  10.100.1.213    db-identity-sau-main-dev-postgresql.fastorder.com


[OK]   PostgreSQL 'identity-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key \
        host=db-identity-sau-main-dev-postgresql-coordinator port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.213
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-coordinator
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: ce097707-5ce5-40c8-a941-01512555cab8
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 📦 03 role...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 11:37:15 UTC] USER=www-data EUID=0 PID=2020474 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    debezium_user
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   debezium_user
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:37:39 UTC] USER=www-data EUID=0 PID=2020736 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-debezium_user
[2026-01-02 11:37:39 UTC] USER=www-data EUID=0 PID=2020745 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-01-02 11:37:39 UTC] USER=www-data EUID=0 PID=2020754 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-debezium_user/ra_root.key
[2026-01-02 11:37:39 UTC] USER=www-data EUID=0 PID=2020763 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-01-02 11:37:39 UTC] USER=www-data EUID=0 PID=2020774 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:37:40 UTC] USER=www-data EUID=0 PID=2020790 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:37:40 UTC] USER=www-data EUID=0 PID=2020799 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:37:40 UTC] USER=www-data EUID=0 PID=2020808 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 11:37:40 UTC] USER=www-data EUID=0 PID=2020817 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 11:37:40 UTC] USER=www-data EUID=0 PID=2020826 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:37:40 UTC] USER=www-data EUID=0 PID=2020835 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:37:40 UTC] USER=www-data EUID=0 PID=2020844 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 11:37:40 UTC] USER=www-data EUID=0 PID=2020853 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 11:37:40 UTC] USER=www-data EUID=0 PID=2020862 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 11:37:40 UTC] USER=www-data EUID=0 PID=2020871 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:37:40 UTC] USER=www-data EUID=0 PID=2020880 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:37:40 UTC] USER=www-data EUID=0 PID=2020889 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 11:37:40 UTC] USER=www-data EUID=0 PID=2020898 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 11:37:40 UTC] USER=www-data EUID=0 PID=2020907 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 11:37:40 UTC] USER=www-data EUID=0 PID=2020917 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:37:40 UTC] USER=www-data EUID=0 PID=2020926 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2020952 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2020961 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2020970 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2020979 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2020988 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2020997 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2021007 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2021017 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2021026 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2021035 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2021044 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2021054 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2021064 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2021075 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2021084 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2021102 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2021111 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2021120 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2021129 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2021140 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:37:41 UTC] USER=www-data EUID=0 PID=2021149 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021171 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021182 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021191 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021200 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021209 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021218 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021227 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021237 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021246 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021255 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021264 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021273 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021283 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021293 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021303 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021312 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021323 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021332 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021341 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021350 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021359 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021368 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021377 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021387 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021398 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:37:42 UTC] USER=www-data EUID=0 PID=2021408 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:37:43 UTC] USER=www-data EUID=0 PID=2021418 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:37:43 UTC] USER=www-data EUID=0 PID=2021427 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 11:37:43 UTC] USER=www-data EUID=0 PID=2021436 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 11:37:43 UTC] USER=www-data EUID=0 PID=2021445 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 11:37:43 UTC] USER=www-data EUID=0 PID=2021454 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:37:43 UTC] USER=www-data EUID=0 PID=2021463 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:37:43 UTC] USER=www-data EUID=0 PID=2021472 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:37:43 UTC] USER=www-data EUID=0 PID=2021481 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: debezium_user
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U debezium_user -d postgres

✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: fastorder_admin_gd
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: coordinator/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-identity-sau-main-dev-postgresql.fastorder.com
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    fastorder_admin_gd
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   fastorder_admin_gd
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:37:59 UTC] USER=www-data EUID=0 PID=2022236 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-fastorder_admin_gd
[2026-01-02 11:37:59 UTC] USER=www-data EUID=0 PID=2022245 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-02 11:37:59 UTC] USER=www-data EUID=0 PID=2022254 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
[2026-01-02 11:37:59 UTC] USER=www-data EUID=0 PID=2022263 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-02 11:37:59 UTC] USER=www-data EUID=0 PID=2022272 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022287 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022296 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022305 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022314 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022323 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022333 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022344 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022353 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022362 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022371 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022380 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022389 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022398 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022407 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022416 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022425 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022437 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:38:00 UTC] USER=www-data EUID=0 PID=2022446 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022472 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022481 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022490 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022499 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022510 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022519 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022528 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022537 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022546 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022555 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022565 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022576 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022586 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022595 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022604 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022625 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022642 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022651 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 11:38:01 UTC] USER=www-data EUID=0 PID=2022660 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022669 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022678 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022687 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022697 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022709 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022719 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022729 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022738 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022747 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022756 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022765 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022774 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022783 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022794 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022803 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022812 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022822 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022832 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022843 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022852 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:38:02 UTC] USER=www-data EUID=0 PID=2022861 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:38:03 UTC] USER=www-data EUID=0 PID=2022870 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 11:38:03 UTC] USER=www-data EUID=0 PID=2022879 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 11:38:03 UTC] USER=www-data EUID=0 PID=2022888 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 11:38:03 UTC] USER=www-data EUID=0 PID=2022906 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:38:03 UTC] USER=www-data EUID=0 PID=2022917 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 11:38:03 UTC] USER=www-data EUID=0 PID=2022926 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 11:38:03 UTC] USER=www-data EUID=0 PID=2022936 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:38:03 UTC] USER=www-data EUID=0 PID=2022955 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:38:03 UTC] USER=www-data EUID=0 PID=2022964 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 11:38:03 UTC] USER=www-data EUID=0 PID=2022973 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 11:38:03 UTC] USER=www-data EUID=0 PID=2022982 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 11:38:03 UTC] USER=www-data EUID=0 PID=2022991 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:38:03 UTC] USER=www-data EUID=0 PID=2023002 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:38:04 UTC] USER=www-data EUID=0 PID=2023020 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: fastorder_admin_gd
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-identity-sau-main-dev-coordinator:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_identity_sau_main_dev_db already exists, skipping creation
[2026-01-02 11:38:04 UTC] USER=www-data EUID=0 PID=2023079 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-02 11:38:04 UTC] USER=www-data EUID=0 PID=2023114 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

=== Pre-flight Checks ===
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
✓ AWS Secrets Manager accessible

=== Retrieving Credentials from AWS ===
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
/opt/fastorder/bash/infra_core/cache.sh: line 145: /var/cache/secrets/fastorder_db_identity_sau_main_dev_postgresql_coordinator_fastorder_admin_gd.cache.tmp.2023129: Permission denied
✅ Retrieved from secrets manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials loaded for coordinator/fastorder_admin_gd: fastorder_admin_gd@db-identity-sau-main-dev-postgresql.fastorder.com:5432/fastorder_identity_sau_main_dev_db
✓ Credentials retrieved: fastorder_admin_gd@db-identity-sau-main-dev-postgresql.fastorder.com:5432/fastorder_identity_sau_main_dev_db
╔════════════════════════════════════════════╗
║  PostgreSQL Test Suite (AWS Secrets MGR)  ║
╚════════════════════════════════════════════╝

=== PostgreSQL Authentication Test ===
✗ PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-identity-sau-main-dev-postgresql.fastorder.com" (10.100.1.213), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 11:38:15 UTC] USER=www-data EUID=0 PID=2023335 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : coordinator
  PG HOST     : db-identity-sau-main-dev-postgresql.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_identity_sau_main_dev_db
  SCHEMA      : auth
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
  DNS → 10.100.1.213
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User debezium_user does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: debezium_user
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: coordinator/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-01-02 11:38:31 UTC] USER=www-data EUID=0 PID=2023700 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
[2026-01-02 11:38:31 UTC] USER=www-data EUID=0 PID=2023728 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d fastorder_identity_sau_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=coordinator IDENTIFIER_PARENT=coordinator
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║                    IAM Database Schema Initialization                       ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] 🟢 Starting IAM schema provisioning...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

[INFO] 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: core
  Core Identity Directory (tenants, realms, identities, devices, MFA)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [1/20]: core/01-tenant
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
🔧 Installing extensions...
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
   Creating reference table: core.tenant
 create_reference_table 
------------------------
 
(1 row)

✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[OK] Table core/01-tenant initialized

[INFO] 🔸 Table [2/20]: core/02-realm
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.realm$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_realm_updated" for relation "core.realm" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[OK] Table core/02-realm initialized

[INFO] 🔸 Table [3/20]: core/03-identity
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_identity_updated" for relation "core.identity" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[OK] Table core/03-identity initialized

[INFO] 🔸 Table [4/20]: core/04-device
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.device$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[OK] Table core/04-device initialized

[INFO] 🔸 Table [5/20]: core/05-identity_account
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity_account$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_identity_account_updated" for relation "core.identity_account" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[OK] Table core/05-identity_account initialized

[INFO] 🔸 Table [6/20]: core/06-identity_mfa
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity_mfa$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[OK] Table core/06-identity_mfa initialized

[INFO] 🔸 Table [7/20]: core/07-external_idp_link
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.external_idp_link$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[OK] Table core/07-external_idp_link initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: policy
  RBAC/ABAC Authorization (clients, roles, permissions, policies)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [8/20]: policy/01-client
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.client$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
NOTICE:  trigger "tr_client_updated" for relation "policy.client" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[OK] Table policy/01-client initialized

[INFO] 🔸 Table [9/20]: policy/02-resource
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.resource$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[OK] Table policy/02-resource initialized

[INFO] 🔸 Table [10/20]: policy/03-scope
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.scope$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_scope_updated" for relation "policy.scope" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[OK] Table policy/03-scope initialized

[INFO] 🔸 Table [11/20]: policy/04-permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.permission$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_permission_updated" for relation "policy.permission" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[OK] Table policy/04-permission initialized

[INFO] 🔸 Table [12/20]: policy/05-role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.role$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_role_updated" for relation "policy.role" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[OK] Table policy/05-role initialized

[INFO] 🔸 Table [13/20]: policy/06-role_permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.role_permission$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[OK] Table policy/06-role_permission initialized

[INFO] 🔸 Table [14/20]: policy/07-identity_role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.identity_role$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[OK] Table policy/07-identity_role initialized

[INFO] 🔸 Table [15/20]: policy/08-policy_rule
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.policy_rule$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_policy_rule_updated" for relation "policy.policy_rule" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[OK] Table policy/08-policy_rule initialized

[INFO] 🔸 Table [16/20]: policy/09-api_key
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.api_key$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[OK] Table policy/09-api_key initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: audit
  Audit & Risk Logging (auth events, admin actions, risk decisions)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [17/20]: audit/01-auth_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[OK] Table audit/01-auth_event initialized

[INFO] 🔸 Table [18/20]: audit/02-admin_action
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[OK] Table audit/02-admin_action initialized

[INFO] 🔸 Table [19/20]: audit/03-risk_decision
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[OK] Table audit/03-risk_decision initialized

[INFO] 🔸 Table [20/20]: audit/04-consent_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[OK] Table audit/04-consent_event initialized


════════════════════════════════════════════════════════════════════════════
[OK] ✅ IAM Schema Initialization Complete!
[OK] All 20 tables initialized successfully

Schemas created:
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

Design highlights:
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

════════════════════════════════════════════════════════════════════════════

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=coordinator IDENTIFIER_PARENT=coordinator
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: identity/run.sh started for SERVICE=identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity
🔍 DEBUG_CHECKPOINT_A3: SERVICE_ROOT exists, discovering table folders
🔍 DEBUG_CHECKPOINT_A4: Found subfolder: auth
🔍 DEBUG_CHECKPOINT_A4b: Checking for nested schema layout in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth
🔍 DEBUG_CHECKPOINT_A4c: Found nested steps dir: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps (display: auth/login)
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: auth/login|/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
[INFO] 📚 Detected grouped table folders under identity/: auth/login

🔍 DEBUG_CHECKPOINT_A7: Current IDENTIFIER=coordinator
🔍 DEBUG_CHECKPOINT_A8_PROCEED: Processing tables on coordinator/main node
🔍 DEBUG_CHECKPOINT_A9: Processing table: auth/login at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
[INFO] 🔸 Table group: auth/login
🔍 DEBUG_CHECKPOINT_A10: About to run numbered steps for table: auth/login
🔍 DEBUG_CHECKPOINT_B1: run_all_numbered_steps_in_dir called for dir=/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps table=auth/login
🔍 DEBUG_CHECKPOINT_B2: Found 1 numbered steps: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B3: About to run step: 01-init-schema.sh
Ab substep 0 compelete start
[DEBUG] Tracking substep start: steps/01-install/steps/identity/auth/login/01-init-schema (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
Ab substep 0 compelete start
[INFO] 📦 01 init schema...
Ab substep 1 compelete start
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing auth.login_account table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating auth schema...
NOTICE:  schema "auth" already exists, skipping
CREATE SCHEMA
✅ Schema created
🔧 Creating account_status ENUM...
DO
✅ ENUM created
🔧 Creating auth.login_account table...
CREATE TABLE
✅ Table created (Citus-compatible with region_hint in all constraints)
🔧 Creating indexes...
CREATE INDEX
CREATE INDEX
✅ Indexes created
🔧 Creating Citus REFERENCE table for CDC compatibility...
 create_reference_table 
------------------------
 
(1 row)

✅ Table created as REFERENCE table (replicated to all nodes)
   CDC via Debezium will work correctly on coordinator
🎉 Schema initialization complete for fastorder_identity_sau_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

📊 Registering environment in monitoring database (obs schema)...
   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
   Resource IP: 142.93.238.16
⚠️  Could not connect to monitoring database, skipping registration
   You can manually register later using:
   /opt/fastorder/bash/scripts/env_app_setup/setup/04-postgresql/steps/register-authN-af-aaaa1-dev.sh

==========================================
✅ Schema initialization complete!
==========================================
Ab substep 1 compelete end
Ab substep 2 compelete start
Ab substep 2 compelete end

🔍 DEBUG_CHECKPOINT_B4: Completed step: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B5: All numbered steps completed for /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A11: Completed numbered steps for table: auth/login
compeleted here

🔍 DEBUG_CHECKPOINT_A12: All tables processed
End of 04-postgresql/steps/01-install/steps/identity/run.sh

✓ ✅ Coordinator setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up 1 worker(s) (Citus data nodes)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up worker: worker-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-02 11:40:30 UTC] USER=unknown EUID=33 PID=2027775 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 11:40:30 UTC] USER=unknown EUID=33 PID=2027783 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 11:40:30 UTC] USER=unknown EUID=33 PID=2027790 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-02 11:40:30 UTC] USER=unknown EUID=33 PID=2027797 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 11:40:30 UTC] USER=unknown EUID=33 PID=2027804 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-02 11:40:30 UTC] USER=unknown EUID=33 PID=2027811 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38 JOB_UUID=25733ea8-d48a-48b5-970c-0036cccf455b

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.214
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01...
[INFO]   db-identity-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.214

[INFO]   ✅ db-identity-sau-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.214    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 11:40:34 UTC] USER=www-data EUID=0 PID=2027939 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:40:34 UTC] USER=www-data EUID=0 PID=2027948 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-02 11:40:34 UTC] USER=www-data EUID=0 PID=2027958 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-2027903
[2026-01-02 11:40:34 UTC] USER=www-data EUID=0 PID=2027967 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-2027903/ra_root.crt
[2026-01-02 11:40:34 UTC] USER=www-data EUID=0 PID=2027976 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-2027903/ra_root.key
[2026-01-02 11:40:34 UTC] USER=www-data EUID=0 PID=2027985 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-2027903/ra_root.crt
[2026-01-02 11:40:34 UTC] USER=www-data EUID=0 PID=2027994 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-2027903/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[2026-01-02 11:40:38 UTC] USER=www-data EUID=0 PID=2028046 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2027903/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 11:40:38 UTC] USER=www-data EUID=0 PID=2028055 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2027903/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 11:40:38 UTC] USER=www-data EUID=0 PID=2028065 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 11:40:38 UTC] USER=www-data EUID=0 PID=2028075 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2027903/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:38 UTC] USER=www-data EUID=0 PID=2028084 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:38 UTC] USER=www-data EUID=0 PID=2028093 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:38 UTC] USER=www-data EUID=0 PID=2028102 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 11:40:38 UTC] USER=www-data EUID=0 PID=2028113 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 11:40:38 UTC] USER=www-data EUID=0 PID=2028122 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 11:40:38 UTC] USER=www-data EUID=0 PID=2028133 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 11:40:38 UTC] USER=www-data EUID=0 PID=2028142 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01
Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:40:39 UTC] USER=www-data EUID=0 PID=2028216 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-01-02 11:40:39 UTC] USER=www-data EUID=0 PID=2028225 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 11:40:39 UTC] USER=www-data EUID=0 PID=2028234 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-01-02 11:40:39 UTC] USER=www-data EUID=0 PID=2028243 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 11:40:39 UTC] USER=www-data EUID=0 PID=2028252 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:40:40 UTC] USER=www-data EUID=0 PID=2028267 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:40:40 UTC] USER=www-data EUID=0 PID=2028276 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:40:40 UTC] USER=www-data EUID=0 PID=2028285 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 11:40:40 UTC] USER=www-data EUID=0 PID=2028294 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 11:40:40 UTC] USER=www-data EUID=0 PID=2028303 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:40 UTC] USER=www-data EUID=0 PID=2028312 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:40:40 UTC] USER=www-data EUID=0 PID=2028321 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 11:40:40 UTC] USER=www-data EUID=0 PID=2028330 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 11:40:40 UTC] USER=www-data EUID=0 PID=2028339 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 11:40:40 UTC] USER=www-data EUID=0 PID=2028348 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:40 UTC] USER=www-data EUID=0 PID=2028368 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 11:40:40 UTC] USER=www-data EUID=0 PID=2028377 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 11:40:40 UTC] USER=www-data EUID=0 PID=2028386 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 11:40:40 UTC] USER=www-data EUID=0 PID=2028395 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:40 UTC] USER=www-data EUID=0 PID=2028404 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:40 UTC] USER=www-data EUID=0 PID=2028431 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028440 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028449 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028458 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028467 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028478 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028487 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028496 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028505 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028514 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028523 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028533 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028552 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028561 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028570 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028579 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028588 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028597 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028607 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028616 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028625 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028634 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028644 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028654 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028663 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:40:41 UTC] USER=www-data EUID=0 PID=2028672 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028681 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028690 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028699 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028709 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028718 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028727 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028736 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028745 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028755 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028765 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028774 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028783 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028792 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028801 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028810 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028819 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028828 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028837 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028847 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028858 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028869 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028879 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028888 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028897 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028906 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 11:40:42 UTC] USER=www-data EUID=0 PID=2028917 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 11:40:43 UTC] USER=www-data EUID=0 PID=2028926 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:40:43 UTC] USER=www-data EUID=0 PID=2028935 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:40:43 UTC] USER=www-data EUID=0 PID=2028944 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:40:43 UTC] USER=www-data EUID=0 PID=2028953 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:40:43 UTC] USER=www-data EUID=0 PID=2028998 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-01-02 11:40:43 UTC] USER=www-data EUID=0 PID=2029007 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 11:40:43 UTC] USER=www-data EUID=0 PID=2029016 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-01-02 11:40:44 UTC] USER=www-data EUID=0 PID=2029025 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 11:40:44 UTC] USER=www-data EUID=0 PID=2029034 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:40:44 UTC] USER=www-data EUID=0 PID=2029048 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:40:44 UTC] USER=www-data EUID=0 PID=2029057 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:40:44 UTC] USER=www-data EUID=0 PID=2029066 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 11:40:44 UTC] USER=www-data EUID=0 PID=2029075 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 11:40:44 UTC] USER=www-data EUID=0 PID=2029084 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:44 UTC] USER=www-data EUID=0 PID=2029095 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:40:44 UTC] USER=www-data EUID=0 PID=2029104 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 11:40:44 UTC] USER=www-data EUID=0 PID=2029113 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 11:40:44 UTC] USER=www-data EUID=0 PID=2029122 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 11:40:44 UTC] USER=www-data EUID=0 PID=2029131 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 11:40:44 UTC] USER=www-data EUID=0 PID=2029140 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 11:40:44 UTC] USER=www-data EUID=0 PID=2029149 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029159 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029168 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029177 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029187 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029197 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029206 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029233 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029242 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029251 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029260 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029269 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029278 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029287 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029296 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029305 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029314 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029323 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 11:40:45 UTC] USER=www-data EUID=0 PID=2029333 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029343 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029352 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029361 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029370 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029379 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029388 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029397 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029406 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029415 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029424 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029433 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029443 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029453 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029462 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029471 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029480 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029489 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029500 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029509 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029518 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029527 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029536 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 11:40:46 UTC] USER=www-data EUID=0 PID=2029545 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029555 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029565 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029574 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029583 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029594 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029603 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029612 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029621 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029630 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029639 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029648 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029657 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029667 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029677 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029686 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029695 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029704 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029713 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029724 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029734 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029743 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:40:47 UTC] USER=www-data EUID=0 PID=2029752 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-postgresql environment: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com (10.100.1.214)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.214
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01
[INFO] Data dir:   /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[INFO] Port:       5432
[INFO] Hostname:   db-identity-sau-main-dev-postgresql-worker-01
[2026-01-02 11:40:50 UTC] USER=www-data EUID=0 PID=2029859 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:40:50 UTC] USER=www-data EUID=0 PID=2029882 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:40:50 UTC] USER=www-data EUID=0 PID=2029906 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:40:50 UTC] USER=www-data EUID=0 PID=2029927 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 11:40:51 UTC] USER=www-data EUID=0 PID=2029967 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:40:51 UTC] USER=www-data EUID=0 PID=2029976 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-02 11:40:51 UTC] USER=www-data EUID=0 PID=2029986 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-2029934
[2026-01-02 11:40:51 UTC] USER=www-data EUID=0 PID=2029995 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-2029934/ra_root.crt
[2026-01-02 11:40:51 UTC] USER=www-data EUID=0 PID=2030004 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-2029934/ra_root.key
[2026-01-02 11:40:51 UTC] USER=www-data EUID=0 PID=2030013 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-2029934/ra_root.crt
[2026-01-02 11:40:51 UTC] USER=www-data EUID=0 PID=2030022 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-2029934/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[2026-01-02 11:40:55 UTC] USER=www-data EUID=0 PID=2030405 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2029934/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 11:40:55 UTC] USER=www-data EUID=0 PID=2030414 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2029934/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 11:40:55 UTC] USER=www-data EUID=0 PID=2030423 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 11:40:55 UTC] USER=www-data EUID=0 PID=2030432 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2029934/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:55 UTC] USER=www-data EUID=0 PID=2030441 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:55 UTC] USER=www-data EUID=0 PID=2030450 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:40:55 UTC] USER=www-data EUID=0 PID=2030459 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 11:40:55 UTC] USER=www-data EUID=0 PID=2030470 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 11:40:55 UTC] USER=www-data EUID=0 PID=2030479 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 11:40:55 UTC] USER=www-data EUID=0 PID=2030488 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 11:40:55 UTC] USER=www-data EUID=0 PID=2030497 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 11:40:55 UTC] USER=www-data EUID=0 PID=2030506 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:40:55 UTC] USER=www-data EUID=0 PID=2030515 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01
Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-02 11:40:55 UTC] USER=www-data EUID=0 PID=2030546 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 11:40:55 UTC] USER=www-data EUID=0 PID=2030555 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-02 11:40:55 UTC] USER=www-data EUID=0 PID=2030585 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-worker-01.service
[2026-01-02 11:40:56 UTC] USER=www-data EUID=0 PID=2030613 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-02 11:40:56 UTC] USER=www-data EUID=0 PID=2030645 ACTION=fsop ARGS=rm -rf /var/run/postgresql-identity-sau-main-dev-worker-01
[OK]   No conflicting Postgres left on port 5432
[OK]   Using postgres password from vault provider
[2026-01-02 11:41:00 UTC] USER=www-data EUID=0 PID=2030715 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.zOcOTq
[2026-01-02 11:41:00 UTC] USER=www-data EUID=0 PID=2030782 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 11:41:01 UTC] USER=www-data EUID=0 PID=2030804 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/identity-sau-main-dev
[INFO] Initializing cluster in /var/lib/postgresql/17/identity-sau-main-dev/worker-01 (SCRAM; pwfile)
[WARN] Removing existing data directory: /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 11:41:01 UTC] USER=www-data EUID=0 PID=2030827 ACTION=fsop ARGS=rm -rf /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 11:41:01 UTC] USER=www-data EUID=0 PID=2030849 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 11:41:01 UTC] USER=www-data EUID=0 PID=2030870 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 11:41:01 UTC] USER=www-data EUID=0 PID=2030891 ACTION=fsop ARGS=chmod 700 /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 11:41:01 UTC] USER=www-data EUID=0 PID=2030914 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-02 11:41:01 UTC] USER=www-data EUID=0 PID=2030935 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-02 11:41:02 UTC] USER=www-data EUID=0 PID=2030968 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-02 11:41:02 UTC] USER=www-data EUID=0 PID=2030977 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /var/lib/postgresql/17/identity-sau-main-dev/worker-01 --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.zOcOTq
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/17/identity-sau-main-dev/worker-01 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /var/lib/postgresql/17/identity-sau-main-dev/worker-01 -l logfile start

[OK]   initdb complete
[2026-01-02 11:41:03 UTC] USER=www-data EUID=0 PID=2031020 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.zOcOTq
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-02 11:41:04 UTC] USER=www-data EUID=0 PID=2031070 ACTION=fsop ARGS=cp /tmp/tmp.WSoLLX8lNK /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 11:41:04 UTC] USER=www-data EUID=0 PID=2031091 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 11:41:04 UTC] USER=www-data EUID=0 PID=2031112 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01.service
[2026-01-02 11:41:04 UTC] USER=www-data EUID=0 PID=2031139 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.m8mIwg /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01.service
[2026-01-02 11:41:04 UTC] USER=www-data EUID=0 PID=2031160 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01.service
[OK]   systemd unit written
[2026-01-02 11:41:04 UTC] USER=www-data EUID=0 PID=2031181 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 11:41:04 UTC] USER=www-data EUID=0 PID=2031202 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 11:41:05 UTC] USER=www-data EUID=0 PID=2031225 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-02 11:41:07 UTC] USER=www-data EUID=0 PID=2031345 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-worker-01.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-02 11:41:08 UTC] USER=www-data EUID=0 PID=2031389 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-worker-01.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Provisioning application database and Debezium role (if not exists)...
[INFO] Checking if database fastorder_identity_sau_main_dev_db exists...
[INFO] DB check result: exit_code=0, output='[2026-01-02 11:41:09 UTC] USER=www-data EUID=0 PID=2031548 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_identity_sau_main_dev_db''
[INFO] Creating database fastorder_identity_sau_main_dev_db...
[2026-01-02 11:41:09 UTC] USER=www-data EUID=0 PID=2031572 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_identity_sau_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[OK]   Database fastorder_identity_sau_main_dev_db created
[INFO] Checking if role debezium_user exists...
[INFO] Role check result: exit_code=0, output='[2026-01-02 11:41:09 UTC] USER=www-data EUID=0 PID=2031596 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[INFO] Creating role debezium_user...
[2026-01-02 11:41:10 UTC] USER=www-data EUID=0 PID=2031624 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'vvFljwx1s2EyVkTJ44xWYVPc';
CREATE ROLE
[OK]   Role debezium_user created
[2026-01-02 11:41:10 UTC] USER=www-data EUID=0 PID=2031648 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_identity_sau_main_dev_db" TO debezium_user;
GRANT
[OK]   Application DB (fastorder_identity_sau_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[INFO] Applying connection and memory optimizations...
[INFO] Current settings: max_connections=100, work_mem=4MB
[INFO] Target settings (worker): max_connections=100, work_mem=8MB
[2026-01-02 11:41:10 UTC] USER=www-data EUID=0 PID=2031732 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 100;
ALTER SYSTEM
[2026-01-02 11:41:11 UTC] USER=www-data EUID=0 PID=2031755 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-02 11:41:11 UTC] USER=www-data EUID=0 PID=2031780 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   Settings applied to postgresql.auto.conf
[2026-01-02 11:41:11 UTC] USER=www-data EUID=0 PID=2031795 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01/standby.signal
[INFO] Service recently started (4s ago) - restarting to apply max_connections...
[INFO] Stopping service...
[2026-01-02 11:41:11 UTC] USER=www-data EUID=0 PID=2031817 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-worker-01.service
[INFO] Waiting for port 5432 to be released...
[OK]   Port 5432 released
[INFO] Starting service...
[2026-01-02 11:41:15 UTC] USER=www-data EUID=0 PID=2031862 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-worker-01.service
[2026-01-02 11:41:20 UTC] USER=www-data EUID=0 PID=2031925 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-worker-01.service
[OK]   ✅ Optimization complete: max_connections=100, work_mem=8MB
[OK]   Synchronous replication already configured (synchronous_commit: on)
[INFO] Setting postgres password via centralized script... for worker-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Initial setup: Using password from initdb
✓ PostgreSQL password already set during initdb
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/worker-01
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
[OK]   Password set and persisted
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.214
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01...
[INFO]   db-identity-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.214

[INFO]   ✅ db-identity-sau-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.214    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com


[OK]   PostgreSQL 'identity-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key \
        host=db-identity-sau-main-dev-postgresql-worker-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-worker-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.214
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-worker-01
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 2a8d7237-0c1b-4286-8ffc-cd46f4f7052e
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 📦 03 role...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 11:41:38 UTC] USER=www-data EUID=0 PID=2032332 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    debezium_user
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   debezium_user
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:42:04 UTC] USER=www-data EUID=0 PID=2032918 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-debezium_user
[2026-01-02 11:42:04 UTC] USER=www-data EUID=0 PID=2032927 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-01-02 11:42:04 UTC] USER=www-data EUID=0 PID=2032936 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-debezium_user/ra_root.key
[2026-01-02 11:42:04 UTC] USER=www-data EUID=0 PID=2032945 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-01-02 11:42:04 UTC] USER=www-data EUID=0 PID=2032954 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:42:04 UTC] USER=www-data EUID=0 PID=2032970 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:42:04 UTC] USER=www-data EUID=0 PID=2032979 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:42:04 UTC] USER=www-data EUID=0 PID=2032988 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 11:42:04 UTC] USER=www-data EUID=0 PID=2032997 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 11:42:04 UTC] USER=www-data EUID=0 PID=2033006 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:04 UTC] USER=www-data EUID=0 PID=2033015 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:42:04 UTC] USER=www-data EUID=0 PID=2033024 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 11:42:04 UTC] USER=www-data EUID=0 PID=2033033 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 11:42:04 UTC] USER=www-data EUID=0 PID=2033042 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 11:42:04 UTC] USER=www-data EUID=0 PID=2033051 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033060 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033071 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033080 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033089 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033099 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033108 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033134 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033143 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033152 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033161 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033170 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033180 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033189 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033198 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033207 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033216 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033225 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033235 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033245 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:05 UTC] USER=www-data EUID=0 PID=2033254 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033263 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033273 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033282 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033291 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033300 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033309 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033318 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033327 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033336 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033346 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033356 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033365 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033374 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033383 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033394 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033403 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033412 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033430 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033439 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033448 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033458 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:06 UTC] USER=www-data EUID=0 PID=2033468 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033477 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033488 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033497 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033506 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033515 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033524 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033533 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033542 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033551 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033560 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033570 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033580 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033589 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033598 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033607 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033616 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033625 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033634 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033645 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:42:07 UTC] USER=www-data EUID=0 PID=2033654 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: debezium_user
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U debezium_user -d postgres

🔐 Generating replicator client certificate for worker-01...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:42:08 UTC] USER=www-data EUID=0 PID=2033699 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-02 11:42:08 UTC] USER=www-data EUID=0 PID=2033708 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 11:42:08 UTC] USER=www-data EUID=0 PID=2033717 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-02 11:42:08 UTC] USER=www-data EUID=0 PID=2033726 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 11:42:08 UTC] USER=www-data EUID=0 PID=2033737 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:42:08 UTC] USER=www-data EUID=0 PID=2033752 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:42:08 UTC] USER=www-data EUID=0 PID=2033761 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:42:08 UTC] USER=www-data EUID=0 PID=2033770 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033779 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033799 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033808 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033817 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033826 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033835 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033845 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033855 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033864 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033873 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033882 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033891 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033900 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033909 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033935 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033944 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033953 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033962 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033971 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033980 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033989 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2033998 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:09 UTC] USER=www-data EUID=0 PID=2034007 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034016 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034025 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034035 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034046 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034055 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034064 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034073 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034082 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034091 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034100 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034109 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034118 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034127 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034136 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034146 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034158 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034168 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034177 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034186 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034195 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034205 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034214 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 11:42:10 UTC] USER=www-data EUID=0 PID=2034232 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034241 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034250 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034260 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034272 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034281 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034290 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034299 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034308 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034317 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034326 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034335 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034344 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034353 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034362 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034372 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034382 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034392 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034401 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034410 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034419 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034428 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034437 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034446 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:42:11 UTC] USER=www-data EUID=0 PID=2034455 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

✅ Replicator certificate generated for worker-01
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: fastorder_admin_gd
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: worker-01/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    fastorder_admin_gd
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   fastorder_admin_gd
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:42:29 UTC] USER=www-data EUID=0 PID=2034878 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-fastorder_admin_gd
[2026-01-02 11:42:29 UTC] USER=www-data EUID=0 PID=2034887 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-01-02 11:42:29 UTC] USER=www-data EUID=0 PID=2034896 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
[2026-01-02 11:42:29 UTC] USER=www-data EUID=0 PID=2034905 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-01-02 11:42:29 UTC] USER=www-data EUID=0 PID=2034914 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:42:29 UTC] USER=www-data EUID=0 PID=2034928 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:42:29 UTC] USER=www-data EUID=0 PID=2034937 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:42:29 UTC] USER=www-data EUID=0 PID=2034946 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 11:42:29 UTC] USER=www-data EUID=0 PID=2034955 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 11:42:29 UTC] USER=www-data EUID=0 PID=2034964 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:29 UTC] USER=www-data EUID=0 PID=2034973 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:42:29 UTC] USER=www-data EUID=0 PID=2034982 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 11:42:29 UTC] USER=www-data EUID=0 PID=2034991 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 11:42:29 UTC] USER=www-data EUID=0 PID=2035000 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035009 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035018 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035027 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035036 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035045 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035054 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035063 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035072 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035081 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035109 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035118 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035127 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035136 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035145 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035154 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035163 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035173 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035183 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035192 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035201 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035223 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035232 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:30 UTC] USER=www-data EUID=0 PID=2035241 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035250 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035259 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035268 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035278 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035287 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035296 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035305 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035314 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035324 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035334 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035343 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035352 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035361 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035370 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035379 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035388 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035397 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035406 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035415 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035424 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035434 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035444 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035453 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035462 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035471 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:31 UTC] USER=www-data EUID=0 PID=2035480 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:42:32 UTC] USER=www-data EUID=0 PID=2035489 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 11:42:32 UTC] USER=www-data EUID=0 PID=2035498 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 11:42:32 UTC] USER=www-data EUID=0 PID=2035507 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:42:32 UTC] USER=www-data EUID=0 PID=2035516 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:42:32 UTC] USER=www-data EUID=0 PID=2035525 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 11:42:32 UTC] USER=www-data EUID=0 PID=2035534 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 11:42:32 UTC] USER=www-data EUID=0 PID=2035544 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:42:32 UTC] USER=www-data EUID=0 PID=2035554 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:42:32 UTC] USER=www-data EUID=0 PID=2035563 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:42:32 UTC] USER=www-data EUID=0 PID=2035572 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 11:42:32 UTC] USER=www-data EUID=0 PID=2035581 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 11:42:32 UTC] USER=www-data EUID=0 PID=2035590 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 11:42:32 UTC] USER=www-data EUID=0 PID=2035601 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:42:32 UTC] USER=www-data EUID=0 PID=2035610 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:42:32 UTC] USER=www-data EUID=0 PID=2035619 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:42:32 UTC] USER=www-data EUID=0 PID=2035628 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: fastorder_admin_gd
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-identity-sau-main-dev-worker-01:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_identity_sau_main_dev_db already exists, skipping creation
[2026-01-02 11:42:33 UTC] USER=www-data EUID=0 PID=2035692 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-02 11:42:33 UTC] USER=www-data EUID=0 PID=2035726 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

=== Pre-flight Checks ===
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
✓ AWS Secrets Manager accessible

=== Retrieving Credentials from AWS ===
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
/opt/fastorder/bash/infra_core/cache.sh: line 145: /var/cache/secrets/fastorder_db_identity_sau_main_dev_postgresql_worker-01_fastorder_admin_gd.cache.tmp.2035746: Permission denied
✅ Retrieved from secrets manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials loaded for worker-01/fastorder_admin_gd: fastorder_admin_gd@db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_identity_sau_main_dev_db
✓ Credentials retrieved: fastorder_admin_gd@db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_identity_sau_main_dev_db
╔════════════════════════════════════════════╗
║  PostgreSQL Test Suite (AWS Secrets MGR)  ║
╚════════════════════════════════════════════╝

=== PostgreSQL Authentication Test ===
✗ PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-identity-sau-main-dev-postgresql-worker-01.fastorder.com" (10.100.1.214), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 11:42:42 UTC] USER=www-data EUID=0 PID=2035933 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_identity_sau_main_dev_db
  SCHEMA      : auth
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
  DNS → 10.100.1.214
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User debezium_user does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: debezium_user
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: worker-01/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-01-02 11:42:58 UTC] USER=www-data EUID=0 PID=2036620 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
[2026-01-02 11:42:58 UTC] USER=www-data EUID=0 PID=2036646 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d fastorder_identity_sau_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
🔐 Creating replicator role for worker-01...
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
  DNS → 10.100.1.214
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Creating role: replicator with password
SET
CREATE ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/identity/sau/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🎉 Done.
✅ Replicator role created for worker-01

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=worker-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║                    IAM Database Schema Initialization                       ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] 🟢 Starting IAM schema provisioning...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

[INFO] 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: core
  Core Identity Directory (tenants, realms, identities, devices, MFA)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [1/20]: core/01-tenant
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "pgcrypto" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "citext" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
NOTICE:  schema "utils" already exists, skipping
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
NOTICE:  schema "core" already exists, skipping
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
NOTICE:  relation "tenant" already exists, skipping
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[OK] Table core/01-tenant initialized

[INFO] 🔸 Table [2/20]: core/02-realm
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  relation "realm" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_realm_keycloak_id" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_realm_tenant" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[OK] Table core/02-realm initialized

[INFO] 🔸 Table [3/20]: core/03-identity
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  relation "identity" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_unique_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_unique_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_type" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[OK] Table core/03-identity initialized

[INFO] 🔸 Table [4/20]: core/04-device
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  relation "device" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_device_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_fingerprint" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_trusted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_last_seen" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[OK] Table core/04-device initialized

[INFO] 🔸 Table [5/20]: core/05-identity_account
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  relation "identity_account" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_account_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_lockout" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_last_login" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[OK] Table core/05-identity_account initialized

[INFO] 🔸 Table [6/20]: core/06-identity_mfa
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  relation "identity_mfa" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_mfa_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_active" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[OK] Table core/06-identity_mfa initialized

[INFO] 🔸 Table [7/20]: core/07-external_idp_link
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  relation "external_idp_link" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_external_idp_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_provider" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_email" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[OK] Table core/07-external_idp_link initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: policy
  RBAC/ABAC Authorization (clients, roles, permissions, policies)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [8/20]: policy/01-client
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
NOTICE:  schema "policy" already exists, skipping
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  relation "client" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_client_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_key" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_status" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[OK] Table policy/01-client initialized

[INFO] 🔸 Table [9/20]: policy/02-resource
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  relation "resource" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_resource_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_external" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_owner" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[OK] Table policy/02-resource initialized

[INFO] 🔸 Table [10/20]: policy/03-scope
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  relation "scope" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_scope_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_scope_name" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[OK] Table policy/03-scope initialized

[INFO] 🔸 Table [11/20]: policy/04-permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  relation "permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_permission_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_resource" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[OK] Table policy/04-permission initialized

[INFO] 🔸 Table [12/20]: policy/05-role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  relation "role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_keycloak" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[OK] Table policy/05-role initialized

[INFO] 🔸 Table [13/20]: policy/06-role_permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  relation "role_permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_permission_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_permission_perm" already exists, skipping
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[OK] Table policy/06-role_permission initialized

[INFO] 🔸 Table [14/20]: policy/07-identity_role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  relation "identity_role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_role_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_active" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[OK] Table policy/07-identity_role initialized

[INFO] 🔸 Table [15/20]: policy/08-policy_rule
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  relation "policy_rule" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_policy_rule_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_enabled" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_priority" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[OK] Table policy/08-policy_rule initialized

[INFO] 🔸 Table [16/20]: policy/09-api_key
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  relation "api_key" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_api_key_prefix" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[OK] Table policy/09-api_key initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: audit
  Audit & Risk Logging (auth events, admin actions, risk decisions)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [17/20]: audit/01-auth_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
NOTICE:  schema "audit" already exists, skipping
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
NOTICE:  relation "auth_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_auth_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_result" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_ip" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_session" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_trace" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_risk" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[OK] Table audit/01-auth_event initialized

[INFO] 🔸 Table [18/20]: audit/02-admin_action
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
NOTICE:  relation "admin_action" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_admin_action_actor" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_target" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_trace" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[OK] Table audit/02-admin_action initialized

[INFO] 🔸 Table [19/20]: audit/03-risk_decision
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
NOTICE:  relation "risk_decision" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_risk_decision_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_level" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_decision" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_auth" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[OK] Table audit/03-risk_decision initialized

[INFO] 🔸 Table [20/20]: audit/04-consent_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
NOTICE:  relation "consent_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_consent_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_version" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_granted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  relation "audit.auth_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  relation "audit.auth_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  relation "audit.admin_action_2026_03" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  relation "audit.admin_action_2026_04" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  relation "audit.risk_decision_2026_03" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  relation "audit.risk_decision_2026_04" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  relation "audit.consent_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  relation "audit.consent_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[OK] Table audit/04-consent_event initialized


════════════════════════════════════════════════════════════════════════════
[OK] ✅ IAM Schema Initialization Complete!
[OK] All 20 tables initialized successfully

Schemas created:
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

Design highlights:
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

════════════════════════════════════════════════════════════════════════════

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=worker-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: identity/run.sh started for SERVICE=identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity
🔍 DEBUG_CHECKPOINT_A3: SERVICE_ROOT exists, discovering table folders
🔍 DEBUG_CHECKPOINT_A4: Found subfolder: auth
🔍 DEBUG_CHECKPOINT_A4b: Checking for nested schema layout in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth
🔍 DEBUG_CHECKPOINT_A4c: Found nested steps dir: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps (display: auth/login)
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: auth/login|/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
[INFO] 📚 Detected grouped table folders under identity/: auth/login

🔍 DEBUG_CHECKPOINT_A7: Current IDENTIFIER=coordinator
🔍 DEBUG_CHECKPOINT_A8_PROCEED: Processing tables on coordinator/main node
🔍 DEBUG_CHECKPOINT_A9: Processing table: auth/login at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
[INFO] 🔸 Table group: auth/login
🔍 DEBUG_CHECKPOINT_A10: About to run numbered steps for table: auth/login
🔍 DEBUG_CHECKPOINT_B1: run_all_numbered_steps_in_dir called for dir=/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps table=auth/login
🔍 DEBUG_CHECKPOINT_B2: Found 1 numbered steps: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B3: About to run step: 01-init-schema.sh
Ab substep 0 compelete start
[DEBUG] Tracking substep start: steps/01-install/steps/identity/auth/login/01-init-schema (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
Ab substep 0 compelete start
[INFO] 📦 01 init schema...
Ab substep 1 compelete start
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing auth.login_account table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating auth schema...
NOTICE:  schema "auth" already exists, skipping
CREATE SCHEMA
✅ Schema created
🔧 Creating account_status ENUM...
DO
✅ ENUM created
🔧 Creating auth.login_account table...
NOTICE:  relation "login_account" already exists, skipping
CREATE TABLE
✅ Table created (Citus-compatible with region_hint in all constraints)
🔧 Creating indexes...
NOTICE:  relation "idx_login_account_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_login_account_username" already exists, skipping
CREATE INDEX
✅ Indexes created
ℹ️  Table already registered with Citus
🎉 Schema initialization complete for fastorder_identity_sau_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

📊 Registering environment in monitoring database (obs schema)...
   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
   Resource IP: 142.93.238.16
⚠️  Could not connect to monitoring database, skipping registration
   You can manually register later using:
   /opt/fastorder/bash/scripts/env_app_setup/setup/04-postgresql/steps/register-authN-af-aaaa1-dev.sh

==========================================
✅ Schema initialization complete!
==========================================
Ab substep 1 compelete end
Ab substep 2 compelete start
Ab substep 2 compelete end

🔍 DEBUG_CHECKPOINT_B4: Completed step: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B5: All numbered steps completed for /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A11: Completed numbered steps for table: auth/login
compeleted here

🔍 DEBUG_CHECKPOINT_A12: All tables processed
End of 04-postgresql/steps/01-install/steps/identity/run.sh

✓ ✅ Worker worker-01 setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up standby replicas (1 per worker)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up standby: worker-01-standby-01 (replica of worker-01)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-02 11:44:55 UTC] USER=unknown EUID=33 PID=2040506 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 11:44:55 UTC] USER=unknown EUID=33 PID=2040513 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 11:44:55 UTC] USER=unknown EUID=33 PID=2040520 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-02 11:44:55 UTC] USER=unknown EUID=33 PID=2040527 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 11:44:55 UTC] USER=unknown EUID=33 PID=2040534 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-02 11:44:55 UTC] USER=unknown EUID=33 PID=2040541 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38 JOB_UUID=25733ea8-d48a-48b5-970c-0036cccf455b

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] PostgreSQL IP: 10.100.1.211
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01-standby-01...
[INFO]   db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.211

[INFO]   ✅ db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.211    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  identity-sau-main-dev.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 11:44:59 UTC] USER=www-data EUID=0 PID=2040677 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:44:59 UTC] USER=www-data EUID=0 PID=2040686 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-02 11:44:59 UTC] USER=www-data EUID=0 PID=2040696 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-2040641
[2026-01-02 11:44:59 UTC] USER=www-data EUID=0 PID=2040705 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-2040641/ra_root.crt
[2026-01-02 11:44:59 UTC] USER=www-data EUID=0 PID=2040714 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-2040641/ra_root.key
[2026-01-02 11:44:59 UTC] USER=www-data EUID=0 PID=2040723 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-2040641/ra_root.crt
[2026-01-02 11:44:59 UTC] USER=www-data EUID=0 PID=2040732 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-2040641/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
[2026-01-02 11:45:00 UTC] USER=www-data EUID=0 PID=2040771 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2040641/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 11:45:00 UTC] USER=www-data EUID=0 PID=2040780 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2040641/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 11:45:00 UTC] USER=www-data EUID=0 PID=2040789 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 11:45:00 UTC] USER=www-data EUID=0 PID=2040798 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2040641/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 11:45:00 UTC] USER=www-data EUID=0 PID=2040807 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 11:45:01 UTC] USER=www-data EUID=0 PID=2040816 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 11:45:01 UTC] USER=www-data EUID=0 PID=2040825 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 11:45:01 UTC] USER=www-data EUID=0 PID=2040836 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 11:45:01 UTC] USER=www-data EUID=0 PID=2040845 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 11:45:01 UTC] USER=www-data EUID=0 PID=2040854 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 11:45:01 UTC] USER=www-data EUID=0 PID=2040863 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 11:45:01 UTC] USER=www-data EUID=0 PID=2040872 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:45:01 UTC] USER=www-data EUID=0 PID=2040882 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:identity-sau-main-dev.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01-standby-01
Primary CN:  identity-sau-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=identity-sau-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:45:02 UTC] USER=www-data EUID=0 PID=2041432 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 11:45:02 UTC] USER=www-data EUID=0 PID=2041598 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 11:45:02 UTC] USER=www-data EUID=0 PID=2041660 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:47:08 UTC] USER=www-data EUID=0 PID=2042308 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:47 UTC] USER=www-data EUID=0 PID=2043202 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:49 UTC] USER=www-data EUID=0 PID=2043748 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 11:49:49 UTC] USER=www-data EUID=0 PID=2043829 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 11:49:50 UTC] USER=www-data EUID=0 PID=2043895 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 11:49:50 UTC] USER=www-data EUID=0 PID=2043959 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 11:49:50 UTC] USER=www-data EUID=0 PID=2044089 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 11:49:51 UTC] USER=www-data EUID=0 PID=2044123 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 11:49:51 UTC] USER=www-data EUID=0 PID=2044268 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:52 UTC] USER=www-data EUID=0 PID=2044288 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 11:49:52 UTC] USER=www-data EUID=0 PID=2044336 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 11:49:52 UTC] USER=www-data EUID=0 PID=2044367 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 11:49:52 UTC] USER=www-data EUID=0 PID=2044384 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 11:49:52 UTC] USER=www-data EUID=0 PID=2044430 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:52 UTC] USER=www-data EUID=0 PID=2044512 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:52 UTC] USER=www-data EUID=0 PID=2044547 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:49:52 UTC] USER=www-data EUID=0 PID=2044565 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:49:52 UTC] USER=www-data EUID=0 PID=2044595 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:52 UTC] USER=www-data EUID=0 PID=2044610 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:53 UTC] USER=www-data EUID=0 PID=2044633 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 11:49:53 UTC] USER=www-data EUID=0 PID=2044652 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 11:49:53 UTC] USER=www-data EUID=0 PID=2044673 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 11:49:53 UTC] USER=www-data EUID=0 PID=2044687 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 11:49:53 UTC] USER=www-data EUID=0 PID=2044707 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 11:49:53 UTC] USER=www-data EUID=0 PID=2044731 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 11:49:53 UTC] USER=www-data EUID=0 PID=2044747 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:53 UTC] USER=www-data EUID=0 PID=2044776 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:53 UTC] USER=www-data EUID=0 PID=2044800 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:49:53 UTC] USER=www-data EUID=0 PID=2044822 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:49:53 UTC] USER=www-data EUID=0 PID=2044853 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:53 UTC] USER=www-data EUID=0 PID=2044872 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:53 UTC] USER=www-data EUID=0 PID=2044891 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 11:49:54 UTC] USER=www-data EUID=0 PID=2044916 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 11:49:54 UTC] USER=www-data EUID=0 PID=2044933 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 11:49:54 UTC] USER=www-data EUID=0 PID=2044957 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 11:49:54 UTC] USER=www-data EUID=0 PID=2044980 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 11:49:54 UTC] USER=www-data EUID=0 PID=2044996 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:54 UTC] USER=www-data EUID=0 PID=2045053 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:54 UTC] USER=www-data EUID=0 PID=2045071 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:49:54 UTC] USER=www-data EUID=0 PID=2045082 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:49:54 UTC] USER=www-data EUID=0 PID=2045106 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:55 UTC] USER=www-data EUID=0 PID=2045151 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 11:49:55 UTC] USER=www-data EUID=0 PID=2045169 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 11:49:55 UTC] USER=www-data EUID=0 PID=2045187 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 11:49:55 UTC] USER=www-data EUID=0 PID=2045209 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 11:49:55 UTC] USER=www-data EUID=0 PID=2045229 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 11:49:55 UTC] USER=www-data EUID=0 PID=2045250 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 11:49:55 UTC] USER=www-data EUID=0 PID=2045292 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:55 UTC] USER=www-data EUID=0 PID=2045325 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:55 UTC] USER=www-data EUID=0 PID=2045357 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:49:55 UTC] USER=www-data EUID=0 PID=2045406 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:56 UTC] USER=www-data EUID=0 PID=2045441 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 11:49:56 UTC] USER=www-data EUID=0 PID=2045488 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 11:49:56 UTC] USER=www-data EUID=0 PID=2045517 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 11:49:56 UTC] USER=www-data EUID=0 PID=2045563 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 11:49:56 UTC] USER=www-data EUID=0 PID=2045583 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 11:49:56 UTC] USER=www-data EUID=0 PID=2045623 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:49:56 UTC] USER=www-data EUID=0 PID=2045654 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:49:56 UTC] USER=www-data EUID=0 PID=2045678 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:49:56 UTC] USER=www-data EUID=0 PID=2045698 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 11:49:56 UTC] USER=www-data EUID=0 PID=2045736 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 11:49:56 UTC] USER=www-data EUID=0 PID=2045757 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 11:49:56 UTC] USER=www-data EUID=0 PID=2045802 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:49:57 UTC] USER=www-data EUID=0 PID=2045826 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:49:57 UTC] USER=www-data EUID=0 PID=2045859 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:49:57 UTC] USER=www-data EUID=0 PID=2045878 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:49:58 UTC] USER=www-data EUID=0 PID=2046083 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-01-02 11:49:58 UTC] USER=www-data EUID=0 PID=2046116 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 11:49:58 UTC] USER=www-data EUID=0 PID=2046185 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
[2026-01-02 11:49:58 UTC] USER=www-data EUID=0 PID=2046220 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 11:49:58 UTC] USER=www-data EUID=0 PID=2046244 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:59 UTC] USER=www-data EUID=0 PID=2046461 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:59 UTC] USER=www-data EUID=0 PID=2046484 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:49:59 UTC] USER=www-data EUID=0 PID=2046506 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 11:49:59 UTC] USER=www-data EUID=0 PID=2046525 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 11:49:59 UTC] USER=www-data EUID=0 PID=2046553 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 11:49:59 UTC] USER=www-data EUID=0 PID=2046571 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 11:49:59 UTC] USER=www-data EUID=0 PID=2046585 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 11:49:59 UTC] USER=www-data EUID=0 PID=2046608 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 11:49:59 UTC] USER=www-data EUID=0 PID=2046622 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 11:49:59 UTC] USER=www-data EUID=0 PID=2046639 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 11:49:59 UTC] USER=www-data EUID=0 PID=2046654 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 11:49:59 UTC] USER=www-data EUID=0 PID=2046682 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 11:49:59 UTC] USER=www-data EUID=0 PID=2046699 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:00 UTC] USER=www-data EUID=0 PID=2046730 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 11:50:00 UTC] USER=www-data EUID=0 PID=2046762 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 11:50:00 UTC] USER=www-data EUID=0 PID=2046783 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 11:50:00 UTC] USER=www-data EUID=0 PID=2046801 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 11:50:00 UTC] USER=www-data EUID=0 PID=2046821 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:00 UTC] USER=www-data EUID=0 PID=2046889 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:00 UTC] USER=www-data EUID=0 PID=2046915 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:00 UTC] USER=www-data EUID=0 PID=2046937 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:00 UTC] USER=www-data EUID=0 PID=2046958 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:00 UTC] USER=www-data EUID=0 PID=2046985 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:00 UTC] USER=www-data EUID=0 PID=2047011 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 11:50:01 UTC] USER=www-data EUID=0 PID=2047051 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 11:50:01 UTC] USER=www-data EUID=0 PID=2047080 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 11:50:01 UTC] USER=www-data EUID=0 PID=2047106 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 11:50:01 UTC] USER=www-data EUID=0 PID=2047127 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 11:50:01 UTC] USER=www-data EUID=0 PID=2047149 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 11:50:01 UTC] USER=www-data EUID=0 PID=2047174 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:01 UTC] USER=www-data EUID=0 PID=2047200 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:01 UTC] USER=www-data EUID=0 PID=2047220 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:01 UTC] USER=www-data EUID=0 PID=2047237 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:01 UTC] USER=www-data EUID=0 PID=2047258 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:01 UTC] USER=www-data EUID=0 PID=2047286 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:01 UTC] USER=www-data EUID=0 PID=2047319 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 11:50:02 UTC] USER=www-data EUID=0 PID=2047452 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 11:50:02 UTC] USER=www-data EUID=0 PID=2047510 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 11:50:02 UTC] USER=www-data EUID=0 PID=2047528 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:02 UTC] USER=www-data EUID=0 PID=2047577 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:02 UTC] USER=www-data EUID=0 PID=2047597 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:02 UTC] USER=www-data EUID=0 PID=2047618 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:02 UTC] USER=www-data EUID=0 PID=2047645 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:02 UTC] USER=www-data EUID=0 PID=2047665 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:02 UTC] USER=www-data EUID=0 PID=2047699 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 11:50:03 UTC] USER=www-data EUID=0 PID=2047724 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 11:50:03 UTC] USER=www-data EUID=0 PID=2047754 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 11:50:03 UTC] USER=www-data EUID=0 PID=2047780 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 11:50:03 UTC] USER=www-data EUID=0 PID=2047807 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 11:50:03 UTC] USER=www-data EUID=0 PID=2047841 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:03 UTC] USER=www-data EUID=0 PID=2047873 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:03 UTC] USER=www-data EUID=0 PID=2047889 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:03 UTC] USER=www-data EUID=0 PID=2047912 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:03 UTC] USER=www-data EUID=0 PID=2047930 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:03 UTC] USER=www-data EUID=0 PID=2047946 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:03 UTC] USER=www-data EUID=0 PID=2047965 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 11:50:03 UTC] USER=www-data EUID=0 PID=2047984 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 11:50:04 UTC] USER=www-data EUID=0 PID=2048046 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 11:50:04 UTC] USER=www-data EUID=0 PID=2048096 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 11:50:05 UTC] USER=www-data EUID=0 PID=2048219 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 11:50:05 UTC] USER=www-data EUID=0 PID=2048318 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 11:50:05 UTC] USER=www-data EUID=0 PID=2048344 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:50:05 UTC] USER=www-data EUID=0 PID=2048389 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:50:05 UTC] USER=www-data EUID=0 PID=2048416 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:50:05 UTC] USER=www-data EUID=0 PID=2048451 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 11:50:05 UTC] USER=www-data EUID=0 PID=2048491 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 11:50:05 UTC] USER=www-data EUID=0 PID=2048516 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 11:50:05 UTC] USER=www-data EUID=0 PID=2048537 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:50:05 UTC] USER=www-data EUID=0 PID=2048557 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:50:05 UTC] USER=www-data EUID=0 PID=2048590 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:50:06 UTC] USER=www-data EUID=0 PID=2048621 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:50:07 UTC] USER=www-data EUID=0 PID=2049013 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-02 11:50:07 UTC] USER=www-data EUID=0 PID=2049049 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 11:50:07 UTC] USER=www-data EUID=0 PID=2049067 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-02 11:50:07 UTC] USER=www-data EUID=0 PID=2049104 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 11:50:07 UTC] USER=www-data EUID=0 PID=2049132 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:50:08 UTC] USER=www-data EUID=0 PID=2049330 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:50:08 UTC] USER=www-data EUID=0 PID=2049378 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:50:08 UTC] USER=www-data EUID=0 PID=2049404 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:50:08 UTC] USER=www-data EUID=0 PID=2049458 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 11:50:08 UTC] USER=www-data EUID=0 PID=2049497 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:50:08 UTC] USER=www-data EUID=0 PID=2049515 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:50:08 UTC] USER=www-data EUID=0 PID=2049544 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:50:08 UTC] USER=www-data EUID=0 PID=2049575 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:50:08 UTC] USER=www-data EUID=0 PID=2049610 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:50:08 UTC] USER=www-data EUID=0 PID=2049643 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:50:08 UTC] USER=www-data EUID=0 PID=2049680 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:50:08 UTC] USER=www-data EUID=0 PID=2049726 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:50:08 UTC] USER=www-data EUID=0 PID=2049768 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:50:08 UTC] USER=www-data EUID=0 PID=2049804 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:50:09 UTC] USER=www-data EUID=0 PID=2049862 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:50:09 UTC] USER=www-data EUID=0 PID=2049903 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:50:09 UTC] USER=www-data EUID=0 PID=2049954 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:50:09 UTC] USER=www-data EUID=0 PID=2050000 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:09 UTC] USER=www-data EUID=0 PID=2050144 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:09 UTC] USER=www-data EUID=0 PID=2050201 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:09 UTC] USER=www-data EUID=0 PID=2050278 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:09 UTC] USER=www-data EUID=0 PID=2050396 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:09 UTC] USER=www-data EUID=0 PID=2050439 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:09 UTC] USER=www-data EUID=0 PID=2050474 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:50:09 UTC] USER=www-data EUID=0 PID=2050519 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 11:50:09 UTC] USER=www-data EUID=0 PID=2050552 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:50:09 UTC] USER=www-data EUID=0 PID=2050592 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2050651 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2050719 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2050767 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2050831 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2050862 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2050905 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2050936 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2050964 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2050998 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2051007 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2051016 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2051025 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2051034 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2051043 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2051053 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2051063 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2051072 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2051081 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2051090 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:10 UTC] USER=www-data EUID=0 PID=2051099 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051108 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051117 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051126 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051135 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051144 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051153 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051163 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051173 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051182 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051191 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051200 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051209 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051218 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051227 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051236 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051245 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051263 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051273 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051283 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051292 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051301 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 11:50:11 UTC] USER=www-data EUID=0 PID=2051310 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 11:50:12 UTC] USER=www-data EUID=0 PID=2051319 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 11:50:12 UTC] USER=www-data EUID=0 PID=2051328 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:50:12 UTC] USER=www-data EUID=0 PID=2051337 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:50:12 UTC] USER=www-data EUID=0 PID=2051346 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:50:12 UTC] USER=www-data EUID=0 PID=2051355 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-standby-01-postgresql environment: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com (10.100.1.211)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.211
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] Data dir:   /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[INFO] Port:       5432
[INFO] Hostname:   db-identity-sau-main-dev-postgresql-worker-01-standby-01
[2026-01-02 11:50:14 UTC] USER=www-data EUID=0 PID=2051454 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:14 UTC] USER=www-data EUID=0 PID=2051475 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:14 UTC] USER=www-data EUID=0 PID=2051496 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:14 UTC] USER=www-data EUID=0 PID=2051517 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  identity-sau-main-dev.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 11:50:15 UTC] USER=www-data EUID=0 PID=2051559 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:15 UTC] USER=www-data EUID=0 PID=2051568 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-02 11:50:15 UTC] USER=www-data EUID=0 PID=2051578 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-2051524
[2026-01-02 11:50:15 UTC] USER=www-data EUID=0 PID=2051587 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-2051524/ra_root.crt
[2026-01-02 11:50:15 UTC] USER=www-data EUID=0 PID=2051596 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-2051524/ra_root.key
[2026-01-02 11:50:15 UTC] USER=www-data EUID=0 PID=2051614 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-2051524/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
[2026-01-02 11:50:18 UTC] USER=www-data EUID=0 PID=2051651 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2051524/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 11:50:18 UTC] USER=www-data EUID=0 PID=2051660 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2051524/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 11:50:18 UTC] USER=www-data EUID=0 PID=2051669 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 11:50:18 UTC] USER=www-data EUID=0 PID=2051678 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2051524/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 11:50:18 UTC] USER=www-data EUID=0 PID=2051687 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 11:50:18 UTC] USER=www-data EUID=0 PID=2051696 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 11:50:18 UTC] USER=www-data EUID=0 PID=2051716 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 11:50:18 UTC] USER=www-data EUID=0 PID=2051727 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 11:50:18 UTC] USER=www-data EUID=0 PID=2051737 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 11:50:18 UTC] USER=www-data EUID=0 PID=2051746 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 11:50:18 UTC] USER=www-data EUID=0 PID=2051755 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:18 UTC] USER=www-data EUID=0 PID=2051764 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:identity-sau-main-dev.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01-standby-01
Primary CN:  identity-sau-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=identity-sau-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-02 11:50:18 UTC] USER=www-data EUID=0 PID=2051793 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 11:50:18 UTC] USER=www-data EUID=0 PID=2051802 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 11:50:18 UTC] USER=www-data EUID=0 PID=2051811 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-02 11:50:18 UTC] USER=www-data EUID=0 PID=2051832 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-worker-01-standby-01.service
[2026-01-02 11:50:19 UTC] USER=www-data EUID=0 PID=2051858 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[2026-01-02 11:50:19 UTC] USER=www-data EUID=0 PID=2051889 ACTION=fsop ARGS=rm -rf /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[OK]   No conflicting Postgres left on port 5432
[OK]   Generated new postgres password for initdb
[2026-01-02 11:50:47 UTC] USER=www-data EUID=0 PID=2052068 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.HGqedq
[2026-01-02 11:50:47 UTC] USER=www-data EUID=0 PID=2052089 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.HGqedq
[2026-01-02 11:50:47 UTC] USER=www-data EUID=0 PID=2052112 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 11:50:47 UTC] USER=www-data EUID=0 PID=2052134 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 11:50:47 UTC] USER=www-data EUID=0 PID=2052157 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/identity-sau-main-dev
[INFO] This is a standby. Using pg_basebackup from primary (worker-01)...
[INFO] Setting up replicator role and slot on primary (worker-01)...
ℹ️  Scanning primary for stuck queries from previous failed attempts...
ℹ️  Scanning for stuck queries (timeout: 30s)...
ℹ️  No stuck queries found
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SLOT        : worker_01_standby_01
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
  DNS → 10.100.1.214
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Role replicator already exists, updating password and ensuring REPLICATION privilege
SET
ALTER ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/identity/sau/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🔧 Ensuring replication slot: worker_01_standby_01…
🆕 Creating replication slot worker_01_standby_01
SET
 pg_create_physical_replication_slot 
-------------------------------------
 (worker_01_standby_01,)
(1 row)

✅ Replication slot worker_01_standby_01 created.
🎉 Done.
[OK]   Replicator role and slot created on primary
[INFO] Creating replicator client certificates for connecting to primary (worker-01)...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:50:52 UTC] USER=www-data EUID=0 PID=2052587 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-02 11:50:52 UTC] USER=www-data EUID=0 PID=2052596 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 11:50:52 UTC] USER=www-data EUID=0 PID=2052605 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-02 11:50:52 UTC] USER=www-data EUID=0 PID=2052614 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 11:50:52 UTC] USER=www-data EUID=0 PID=2052623 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:50:52 UTC] USER=www-data EUID=0 PID=2052639 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:50:52 UTC] USER=www-data EUID=0 PID=2052648 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:50:52 UTC] USER=www-data EUID=0 PID=2052657 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:50:52 UTC] USER=www-data EUID=0 PID=2052666 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 11:50:52 UTC] USER=www-data EUID=0 PID=2052675 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:50:52 UTC] USER=www-data EUID=0 PID=2052684 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:50:52 UTC] USER=www-data EUID=0 PID=2052693 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:50:52 UTC] USER=www-data EUID=0 PID=2052704 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:50:52 UTC] USER=www-data EUID=0 PID=2052714 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:50:52 UTC] USER=www-data EUID=0 PID=2052724 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:50:52 UTC] USER=www-data EUID=0 PID=2052733 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:50:52 UTC] USER=www-data EUID=0 PID=2052742 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052751 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052760 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052771 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052780 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052789 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052798 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052824 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052834 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052846 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052855 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052864 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052873 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052882 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052891 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052900 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052909 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052918 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052928 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052938 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052947 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052956 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052965 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:53 UTC] USER=www-data EUID=0 PID=2052974 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2052983 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2052992 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053001 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053010 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053019 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053028 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053038 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053048 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053057 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053066 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053075 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053084 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053093 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053102 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053111 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053120 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053129 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053138 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053148 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053158 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053167 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053176 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053185 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053194 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053203 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053212 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053221 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:50:54 UTC] USER=www-data EUID=0 PID=2053230 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 11:50:55 UTC] USER=www-data EUID=0 PID=2053239 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:50:55 UTC] USER=www-data EUID=0 PID=2053248 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 11:50:55 UTC] USER=www-data EUID=0 PID=2053258 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:50:55 UTC] USER=www-data EUID=0 PID=2053268 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:50:55 UTC] USER=www-data EUID=0 PID=2053277 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:50:55 UTC] USER=www-data EUID=0 PID=2053286 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 11:50:55 UTC] USER=www-data EUID=0 PID=2053295 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 11:50:55 UTC] USER=www-data EUID=0 PID=2053304 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 11:50:55 UTC] USER=www-data EUID=0 PID=2053313 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:50:55 UTC] USER=www-data EUID=0 PID=2053322 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:50:55 UTC] USER=www-data EUID=0 PID=2053331 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:50:55 UTC] USER=www-data EUID=0 PID=2053340 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

[OK]   Replicator certificate created for worker-01 in /home/postgres/
[INFO] Using replicator certificates from primary worker-01...
[2026-01-02 11:50:55 UTC] USER=www-data EUID=0 PID=2053368 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 11:50:55 UTC] USER=www-data EUID=0 PID=2053389 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 11:50:55 UTC] USER=www-data EUID=0 PID=2053410 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[OK]   Replicator certificates verified at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[OK]   root.crt verified at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[INFO] Updating primary pg_hba.conf to allow replication...
[INFO]   Standby IP: 10.100.1.211/32 (standby's source IP)
[INFO]   Primary application IP: 10.100.1.214/32 (for local pg_basebackup)
[INFO]   Primary DNS IP: 10.100.1.214/32 (DNS resolution of db-identity-sau-main-dev-postgresql-worker-01.fastorder.com)
[2026-01-02 11:50:56 UTC] USER=www-data EUID=0 PID=2053440 ACTION=passthru ARGS=grep -qxF # BEGIN standby-replication (managed) /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 11:50:56 UTC] USER=www-data EUID=0 PID=2053484 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.211/32  scram-sha-256 
      $0==begin {inside=1}
      inside && $0==rule {found=1}
      $0==end {inside=0}
      END {exit found?0:1}
     /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 11:50:56 UTC] USER=www-data EUID=0 PID=2053508 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl  replication  replicator  10.100.1.211/32  scram-sha-256 /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 11:50:56 UTC] USER=www-data EUID=0 PID=2053529 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.214/32  scram-sha-256 
        $0==begin {inside=1}
        inside && $0==rule {found=1}
        $0==end {inside=0}
        END {exit found?0:1}
       /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 11:50:56 UTC] USER=www-data EUID=0 PID=2053553 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl  replication  replicator  10.100.1.214/32  scram-sha-256 /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[INFO] Reloading primary PostgreSQL service...
[2026-01-02 11:50:56 UTC] USER=www-data EUID=0 PID=2053574 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
[OK]   Primary pg_hba.conf updated and service reloaded
[WARN] Removing existing data directory: /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 11:50:56 UTC] USER=www-data EUID=0 PID=2053596 ACTION=fsop ARGS=rm -rf /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[INFO] Primary host: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO] Using replicator cert: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[INFO] Using replicator key: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key (PKCS#8 format)
[INFO] Using CA cert: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[INFO] Verifying postgres user can access certificates...
[ERR]  postgres user CANNOT read /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[INFO] File permissions:
lrwxrwxrwx 1 postgres ssl-cert 72 Jan  2 11:50 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt -> /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[INFO] Parent directory permissions:
drwx------ 2 postgres postgres 4096 Jan  2 11:50 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
drwx------ 6 postgres postgres 4096 Jan  2 07:10 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[WARN] Attempting to fix permissions (/usr/local/bin/fastorder-provisioning-wrapper.sh required)...
[INFO] Fixing /home/postgres/ directory...
[2026-01-02 11:50:57 UTC] USER=www-data EUID=0 PID=2053663 ACTION=fsop ARGS=chmod 755 /home/postgres/
[INFO] Fixing /home/postgres/ssl/.postgresql/...
[2026-01-02 11:50:57 UTC] USER=www-data EUID=0 PID=2053684 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/
[INFO] Fixing parent directory: /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:50:57 UTC] USER=www-data EUID=0 PID=2053707 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[INFO] Fixing certificate directory: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 11:50:57 UTC] USER=www-data EUID=0 PID=2053728 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[INFO] Fixing CA certificate: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 11:50:57 UTC] USER=www-data EUID=0 PID=2053749 ACTION=fsop ARGS=chmod 644 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[OK]   Permissions fixed
[OK]   postgres user can now read /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt after permission fix
[2026-01-02 11:50:57 UTC] USER=www-data EUID=0 PID=2053770 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[2026-01-02 11:50:57 UTC] USER=www-data EUID=0 PID=2053791 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[2026-01-02 11:50:58 UTC] USER=www-data EUID=0 PID=2053812 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[INFO] Checking primary database size before pg_basebackup...
[INFO] Total primary database size: 29 MB
[INFO] Estimated transfer time: ~0 minutes (at 10MB/s with compression)
[INFO] Retrieving replicator password from AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/replicator
[OK]   Replicator password retrieved successfully
[INFO] Starting pg_basebackup...
[2026-01-02 11:51:00 UTC] USER=www-data EUID=0 PID=2053884 ACTION=passthru ARGS=sudo -u postgres env PGPASSWORD=qrzga0rZrBWHXjHNfE1t9bdwqo6QF84R PGSSLMODE=verify-full PGSSLCERT=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt PGSSLKEY=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key PGSSLROOTCERT=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /usr/lib/postgresql/17/bin/pg_basebackup -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -p 5432 -U replicator -D /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01 -Fp -Xs -P -R --checkpoint=fast --wal-method=stream --verbose
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/2000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_2053893"
16082/30540 kB (52%), 0/1 tablespace (...er-01-standby-01/base/16384/2840)
30550/30550 kB (100%), 0/1 tablespace (...-01-standby-01/global/pg_control)
30550/30550 kB (100%), 1/1 tablespace                                         
pg_basebackup: write-ahead log end point: 0/2000120
pg_basebackup: waiting for background process to finish streaming ...
pg_basebackup: syncing data to disk ...
pg_basebackup: renaming backup_manifest.tmp to backup_manifest
pg_basebackup: base backup completed
[OK]   pg_basebackup complete
[INFO] Fixing postgresql.auto.conf to use IP-based primary_conninfo (matching golden backup)...
[2026-01-02 11:51:03 UTC] USER=www-data EUID=0 PID=2053909 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-02 11:51:03 UTC] USER=www-data EUID=0 PID=2053933 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-02 11:51:03 UTC] USER=www-data EUID=0 PID=2053956 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-02 11:51:03 UTC] USER=www-data EUID=0 PID=2053965 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[OK]   standby.signal verified and permissions set
[INFO] Fixing postgresql.conf with standby-specific settings...
[WARN] postgresql.conf not found at /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/postgresql.conf
[INFO] Verifying postgresql.auto.conf...
[WARN] postgresql.auto.conf not found - pg_basebackup may have failed
[2026-01-02 11:51:03 UTC] USER=www-data EUID=0 PID=2053988 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.HGqedq
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-02 11:51:03 UTC] USER=www-data EUID=0 PID=2054037 ACTION=fsop ARGS=cp /tmp/tmp.lzTBrxpIda /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/pg_hba.conf
[2026-01-02 11:51:04 UTC] USER=www-data EUID=0 PID=2054060 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/pg_hba.conf
[2026-01-02 11:51:04 UTC] USER=www-data EUID=0 PID=2054081 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01-standby-01.service
[2026-01-02 11:51:04 UTC] USER=www-data EUID=0 PID=2054110 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.M4Gfvi /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01-standby-01.service
[2026-01-02 11:51:04 UTC] USER=www-data EUID=0 PID=2054131 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01-standby-01.service
[OK]   systemd unit written
[2026-01-02 11:51:04 UTC] USER=www-data EUID=0 PID=2054152 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 11:51:04 UTC] USER=www-data EUID=0 PID=2054173 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 11:51:04 UTC] USER=www-data EUID=0 PID=2054194 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-02 11:51:06 UTC] USER=www-data EUID=0 PID=2054308 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-worker-01-standby-01.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-02 11:51:07 UTC] USER=www-data EUID=0 PID=2054348 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-worker-01-standby-01.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Configuring synchronous replication on primary worker-01...
[INFO] Current synchronous_standby_names: ''
[INFO] Initializing synchronous_standby_names with first standby
[INFO] New synchronous_standby_names: 'ANY 1 (worker_01_standby_01)'
[2026-01-02 11:51:08 UTC] USER=www-data EUID=0 PID=2054417 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_commit = on;
ALTER SYSTEM
[2026-01-02 11:51:08 UTC] USER=www-data EUID=0 PID=2054442 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_standby_names = 'ANY 1 (worker_01_standby_01)';
ALTER SYSTEM
[2026-01-02 11:51:08 UTC] USER=www-data EUID=0 PID=2054465 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   ✅ Synchronous replication configured on primary
[OK]      Setting: ANY 1 (worker_01_standby_01)
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Skipping database/role provisioning on standby node (read-only)
[INFO]   Database/roles will be replicated from primary: worker-01
[INFO] Applying connection and memory optimizations...
[INFO] Standby will use primary's max_connections: 100
[INFO] Current settings: max_connections=100, work_mem=8MB
[INFO] Target settings (standby): max_connections=100, work_mem=8MB
[OK]   Connection settings already optimized
[INFO] Skipping password setting - this is a standby (read-only)
[INFO] Use primary's postgres password to connect to this standby
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] PostgreSQL IP: 10.100.1.211
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01-standby-01...
[INFO]   db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.211

[INFO]   ✅ db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.211    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com


[OK]   PostgreSQL 'identity-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key \
        host=db-identity-sau-main-dev-postgresql-worker-01-standby-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-worker-01-standby-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.211
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-worker-01-standby-01
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 8eaa8059-bede-4f71-ae1d-d26590a898da
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 📦 03 role...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 11:51:15 UTC] USER=www-data EUID=0 PID=2054801 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
⚠ This is a PostgreSQL STANDBY (read-only replica)
⚠ Skipping role creation - standby gets roles from primary via replication
⚠ Use the PRIMARY's credentials to connect to this standby


[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=worker-01-standby-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║                    IAM Database Schema Initialization                       ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] 🟢 Starting IAM schema provisioning...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

[INFO] 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: core
  Core Identity Directory (tenants, realms, identities, devices, MFA)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [1/20]: core/01-tenant
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "pgcrypto" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "citext" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
NOTICE:  schema "utils" already exists, skipping
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
NOTICE:  schema "core" already exists, skipping
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
NOTICE:  relation "tenant" already exists, skipping
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[OK] Table core/01-tenant initialized

[INFO] 🔸 Table [2/20]: core/02-realm
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  relation "realm" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_realm_keycloak_id" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_realm_tenant" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[OK] Table core/02-realm initialized

[INFO] 🔸 Table [3/20]: core/03-identity
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  relation "identity" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_unique_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_unique_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_type" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[OK] Table core/03-identity initialized

[INFO] 🔸 Table [4/20]: core/04-device
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  relation "device" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_device_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_fingerprint" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_trusted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_last_seen" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[OK] Table core/04-device initialized

[INFO] 🔸 Table [5/20]: core/05-identity_account
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  relation "identity_account" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_account_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_lockout" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_last_login" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[OK] Table core/05-identity_account initialized

[INFO] 🔸 Table [6/20]: core/06-identity_mfa
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  relation "identity_mfa" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_mfa_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_active" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[OK] Table core/06-identity_mfa initialized

[INFO] 🔸 Table [7/20]: core/07-external_idp_link
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  relation "external_idp_link" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_external_idp_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_provider" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_email" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[OK] Table core/07-external_idp_link initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: policy
  RBAC/ABAC Authorization (clients, roles, permissions, policies)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [8/20]: policy/01-client
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
NOTICE:  schema "policy" already exists, skipping
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  relation "client" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_client_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_key" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_status" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[OK] Table policy/01-client initialized

[INFO] 🔸 Table [9/20]: policy/02-resource
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  relation "resource" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_resource_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_external" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_owner" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[OK] Table policy/02-resource initialized

[INFO] 🔸 Table [10/20]: policy/03-scope
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  relation "scope" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_scope_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_scope_name" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[OK] Table policy/03-scope initialized

[INFO] 🔸 Table [11/20]: policy/04-permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  relation "permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_permission_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_resource" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[OK] Table policy/04-permission initialized

[INFO] 🔸 Table [12/20]: policy/05-role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  relation "role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_keycloak" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[OK] Table policy/05-role initialized

[INFO] 🔸 Table [13/20]: policy/06-role_permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  relation "role_permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_permission_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_permission_perm" already exists, skipping
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[OK] Table policy/06-role_permission initialized

[INFO] 🔸 Table [14/20]: policy/07-identity_role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  relation "identity_role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_role_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_active" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[OK] Table policy/07-identity_role initialized

[INFO] 🔸 Table [15/20]: policy/08-policy_rule
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  relation "policy_rule" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_policy_rule_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_enabled" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_priority" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[OK] Table policy/08-policy_rule initialized

[INFO] 🔸 Table [16/20]: policy/09-api_key
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  relation "api_key" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_api_key_prefix" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[OK] Table policy/09-api_key initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: audit
  Audit & Risk Logging (auth events, admin actions, risk decisions)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [17/20]: audit/01-auth_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
NOTICE:  schema "audit" already exists, skipping
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
NOTICE:  relation "auth_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_auth_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_result" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_ip" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_session" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_trace" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_risk" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[OK] Table audit/01-auth_event initialized

[INFO] 🔸 Table [18/20]: audit/02-admin_action
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
NOTICE:  relation "admin_action" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_admin_action_actor" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_target" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_trace" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[OK] Table audit/02-admin_action initialized

[INFO] 🔸 Table [19/20]: audit/03-risk_decision
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
NOTICE:  relation "risk_decision" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_risk_decision_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_level" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_decision" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_auth" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[OK] Table audit/03-risk_decision initialized

[INFO] 🔸 Table [20/20]: audit/04-consent_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
NOTICE:  relation "consent_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_consent_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_version" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_granted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  relation "audit.auth_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  relation "audit.auth_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  relation "audit.admin_action_2026_03" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  relation "audit.admin_action_2026_04" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  relation "audit.risk_decision_2026_03" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  relation "audit.risk_decision_2026_04" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  relation "audit.consent_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  relation "audit.consent_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[OK] Table audit/04-consent_event initialized


════════════════════════════════════════════════════════════════════════════
[OK] ✅ IAM Schema Initialization Complete!
[OK] All 20 tables initialized successfully

Schemas created:
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

Design highlights:
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

════════════════════════════════════════════════════════════════════════════

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=worker-01-standby-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: identity/run.sh started for SERVICE=identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity
🔍 DEBUG_CHECKPOINT_A3: SERVICE_ROOT exists, discovering table folders
🔍 DEBUG_CHECKPOINT_A4: Found subfolder: auth
🔍 DEBUG_CHECKPOINT_A4b: Checking for nested schema layout in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth
🔍 DEBUG_CHECKPOINT_A4c: Found nested steps dir: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps (display: auth/login)
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: auth/login|/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
[INFO] 📚 Detected grouped table folders under identity/: auth/login

🔍 DEBUG_CHECKPOINT_A7: Current IDENTIFIER=coordinator
🔍 DEBUG_CHECKPOINT_A8_PROCEED: Processing tables on coordinator/main node
🔍 DEBUG_CHECKPOINT_A9: Processing table: auth/login at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
[INFO] 🔸 Table group: auth/login
🔍 DEBUG_CHECKPOINT_A10: About to run numbered steps for table: auth/login
🔍 DEBUG_CHECKPOINT_B1: run_all_numbered_steps_in_dir called for dir=/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps table=auth/login
🔍 DEBUG_CHECKPOINT_B2: Found 1 numbered steps: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B3: About to run step: 01-init-schema.sh
Ab substep 0 compelete start
[DEBUG] Tracking substep start: steps/01-install/steps/identity/auth/login/01-init-schema (RUN_UUID=c84fde75-a6b5-4ab2-9f45-54dfb994bb38)
Ab substep 0 compelete start
[INFO] 📦 01 init schema...
Ab substep 1 compelete start
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing auth.login_account table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating auth schema...
NOTICE:  schema "auth" already exists, skipping
CREATE SCHEMA
✅ Schema created
🔧 Creating account_status ENUM...
DO
✅ ENUM created
🔧 Creating auth.login_account table...
NOTICE:  relation "login_account" already exists, skipping
CREATE TABLE
✅ Table created (Citus-compatible with region_hint in all constraints)
🔧 Creating indexes...
NOTICE:  relation "idx_login_account_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_login_account_username" already exists, skipping
CREATE INDEX
✅ Indexes created
ℹ️  Table already registered with Citus
🎉 Schema initialization complete for fastorder_identity_sau_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

📊 Registering environment in monitoring database (obs schema)...
   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
   Resource IP: 142.93.238.16
⚠️  Could not connect to monitoring database, skipping registration
   You can manually register later using:
   /opt/fastorder/bash/scripts/env_app_setup/setup/04-postgresql/steps/register-authN-af-aaaa1-dev.sh

==========================================
✅ Schema initialization complete!
==========================================
Ab substep 1 compelete end
Ab substep 2 compelete start
Ab substep 2 compelete end

🔍 DEBUG_CHECKPOINT_B4: Completed step: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B5: All numbered steps completed for /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A11: Completed numbered steps for table: auth/login
compeleted here

🔍 DEBUG_CHECKPOINT_A12: All tables processed
End of 04-postgresql/steps/01-install/steps/identity/run.sh

✓ ✅ Standby worker-01-standby-01 setup completed

✓ ✅ PostgreSQL installation completed
[INFO] Discovering additional setup steps...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 02-pg-bouncer.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up PgBouncer connection pooling...
[2026-01-02 11:52:57 UTC] USER=www-data EUID=0 PID=2058227 ACTION=fsop ARGS=rm -f /tmp/pgbouncer-ip.service /tmp/pgbouncer.service
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] Checking for existing PgBouncer application environment in topology …
[OK]   Using existing PgBouncer environment:
[INFO]   IP:     10.100.1.204
[INFO]   FQDN:   db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Domain: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO] Ensuring /etc/hosts entry for db-identity-sau-main-dev-postgresql-bouncer.fastorder.com …
[OK]   /etc/hosts already contains entry for db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[WARN] IP 10.100.1.204 is assigned to multiple interfaces:
    inet 10.100.1.103/32 scope global lo
       valid_lft forever preferred_lft forever
    inet 10.100.1.204/32 scope global lo:pgbouncer
--
    inet 10.100.1.214/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.100.1.204/32 scope global eth0:pgbouncer
[WARN] This may cause routing issues
[INFO] Final verification of /etc/hosts entry for db-identity-sau-main-dev-postgresql-bouncer.fastorder.com …
[OK]   /etc/hosts correctly maps db-identity-sau-main-dev-postgresql-bouncer.fastorder.com to 10.100.1.204
[OK]   PgBouncer IP 10.100.1.204 already correctly bound to lo:pgbouncer
[2026-01-02 11:52:58 UTC] USER=www-data EUID=0 PID=2058307 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 11:52:59 UTC] USER=www-data EUID=0 PID=2058400 ACTION=passthru ARGS=systemctl restart pgbouncer-ip@identity-sau-main-dev.service
[2026-01-02 11:52:59 UTC] USER=www-data EUID=0 PID=2058411 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer-ip@identity-sau-main-dev.service
[OK]   pgbouncer-ip@identity-sau-main-dev.service is active
[2026-01-02 11:52:59 UTC] USER=www-data EUID=0 PID=2058435 ACTION=fsop ARGS=mkdir -p /etc/pgbouncer/identity-sau-main-dev
[2026-01-02 11:52:59 UTC] USER=www-data EUID=0 PID=2058444 ACTION=fsop ARGS=mkdir -p /run/pgbouncer/identity-sau-main-dev
[2026-01-02 11:52:59 UTC] USER=www-data EUID=0 PID=2058453 ACTION=fsop ARGS=mkdir -p /var/log/pgbouncer/identity-sau-main-dev
[2026-01-02 11:53:00 UTC] USER=www-data EUID=0 PID=2058462 ACTION=fsop ARGS=chmod 750 /etc/pgbouncer/identity-sau-main-dev
[2026-01-02 11:53:00 UTC] USER=www-data EUID=0 PID=2058471 ACTION=fsop ARGS=chmod 750 /run/pgbouncer/identity-sau-main-dev
[2026-01-02 11:53:00 UTC] USER=www-data EUID=0 PID=2058480 ACTION=fsop ARGS=chmod 750 /var/log/pgbouncer/identity-sau-main-dev
[2026-01-02 11:53:00 UTC] USER=www-data EUID=0 PID=2058489 ACTION=fsop ARGS=chown root:postgres /etc/pgbouncer/identity-sau-main-dev
[2026-01-02 11:53:00 UTC] USER=www-data EUID=0 PID=2058498 ACTION=fsop ARGS=chown postgres:postgres /run/pgbouncer/identity-sau-main-dev
[2026-01-02 11:53:00 UTC] USER=www-data EUID=0 PID=2058507 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbouncer/identity-sau-main-dev
[INFO] Generating pgbouncer_admin client certificates...
[INFO] ⏳ This may take 30-60 seconds...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    pgbouncer_admin
Identifier:  pgbouncer
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        pgbouncer
  User (CN):   pgbouncer_admin
  Hostname:    db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:53:00 UTC] USER=www-data EUID=0 PID=2058541 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-pgbouncer-pgbouncer_admin
[2026-01-02 11:53:00 UTC] USER=www-data EUID=0 PID=2058550 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-02 11:53:00 UTC] USER=www-data EUID=0 PID=2058559 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
[2026-01-02 11:53:00 UTC] USER=www-data EUID=0 PID=2058568 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-02 11:53:00 UTC] USER=www-data EUID=0 PID=2058577 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = pgbouncer_admin
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:01 UTC] USER=www-data EUID=0 PID=2058591 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:01 UTC] USER=www-data EUID=0 PID=2058600 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:01 UTC] USER=www-data EUID=0 PID=2058609 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 11:53:01 UTC] USER=www-data EUID=0 PID=2058618 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 11:53:01 UTC] USER=www-data EUID=0 PID=2058627 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 11:53:01 UTC] USER=www-data EUID=0 PID=2058636 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 11:53:01 UTC] USER=www-data EUID=0 PID=2058645 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 11:53:01 UTC] USER=www-data EUID=0 PID=2058654 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 11:53:01 UTC] USER=www-data EUID=0 PID=2058663 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 11:53:01 UTC] USER=www-data EUID=0 PID=2058672 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 11:53:01 UTC] USER=www-data EUID=0 PID=2058681 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 11:53:01 UTC] USER=www-data EUID=0 PID=2058690 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 11:53:01 UTC] USER=www-data EUID=0 PID=2058699 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:01 UTC] USER=www-data EUID=0 PID=2058708 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 11:53:01 UTC] USER=www-data EUID=0 PID=2058717 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 11:53:01 UTC] USER=www-data EUID=0 PID=2058726 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 11:53:01 UTC] USER=www-data EUID=0 PID=2058735 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058756 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058782 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058791 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058800 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058809 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058818 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058827 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058836 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058845 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058854 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058863 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058872 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058882 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058894 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058903 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058912 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058921 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058930 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058939 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058948 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058957 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058966 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058975 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058984 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 11:53:02 UTC] USER=www-data EUID=0 PID=2058994 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059004 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059013 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059022 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059031 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059040 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059049 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059058 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059067 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059076 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059085 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059094 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059104 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059116 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059125 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059134 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059143 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059152 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059161 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059170 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059179 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059188 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059197 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059206 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059216 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059226 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:53:03 UTC] USER=www-data EUID=0 PID=2059235 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059244 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059253 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059262 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059271 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059280 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059289 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059298 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: pgbouncer_admin
Node: pgbouncer
FQDN: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-bouncer.fastorder.com -U pgbouncer_admin -d postgres

[OK]   mTLS client certificate present: /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[INFO] Creating symlinks to canonical certificates in /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend...
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059314 ACTION=fsop ARGS=mkdir -p /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059323 ACTION=fsop ARGS=mkdir -p /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059332 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059341 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059350 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt
[INFO] Creating coordinator CA symlink for PostgreSQL server verification...
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059359 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] Verifying canonical certificate permissions...
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059368 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059377 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059386 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059395 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[OK]   Backend certificate symlinks created in /etc/ssl
[OK]   Coordinator CA symlink created for server verification
[OK]   Certificates already in canonical location - no symlinks needed
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059406 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.crt
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059415 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.key
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059424 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059434 ACTION=fsop ARGS=test -r /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] PgBouncer will use PostgreSQL coordinator CA: /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[OK]   PostgreSQL coordinator at db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432 is reachable
[INFO] Dumping SCRAM secrets from coordinator for PgBouncer auth_file …
[2026-01-02 11:53:04 UTC] USER=www-data EUID=0 PID=2059453 ACTION=fsop ARGS=cp /tmp/tmp.FWRYxeXjgx /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-02 11:53:05 UTC] USER=www-data EUID=0 PID=2059462 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-02 11:53:05 UTC] USER=www-data EUID=0 PID=2059471 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[OK]   Auth file written: /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[INFO] Retrieved password from vault for pgbouncer_admin
[INFO] Ensuring PgBouncer admin role 'pgbouncer_admin' exists in Postgres (coordinator) …
[OK]   Role pgbouncer_admin created/updated successfully
[SECRETS] Setting credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/pgbouncer_admin
✓ [SECRETS] Credentials updated in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/pgbouncer_admin
[INFO] ✅ PgBouncer admin password stored in centralized secrets vault
[INFO] Re-fetching SCRAM secrets after role creation to ensure pgbouncer_admin is included …
[2026-01-02 11:53:13 UTC] USER=www-data EUID=0 PID=2059526 ACTION=fsop ARGS=cp /tmp/tmp.Z4fYsuclKM /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-02 11:53:13 UTC] USER=www-data EUID=0 PID=2059535 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-02 11:53:13 UTC] USER=www-data EUID=0 PID=2059544 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[OK]   Auth file updated with pgbouncer_admin SCRAM hash
[INFO] Auth file contains [2026-01-02 11:53:13 UTC] USER=www-data EUID=0 PID=2059556 ACTION=passthru ARGS=bash -c wc -l < '/etc/pgbouncer/identity-sau-main-dev/userlist.txt'
4 user(s)
[OK]   Admin 'pgbouncer_admin' password generated and saved
[INFO] Configuring PostgreSQL to prevent Citus metadata sync hangs...
ALTER ROLE
[OK]   Disabled Citus metadata sync for pgbouncer_admin
[INFO] Verifying application database fastorder_identity_sau_main_dev_db exists...
[OK]   ✓ Database fastorder_identity_sau_main_dev_db exists
[INFO] Granting permissions to pgbouncer_admin on fastorder_identity_sau_main_dev_db...
GRANT
[OK]   ✓ Granted CONNECT on fastorder_identity_sau_main_dev_db to pgbouncer_admin
GRANT
[OK]   ✓ Granted USAGE on schema public to pgbouncer_admin
GRANT
[OK]   ✓ Granted SELECT on all tables to pgbouncer_admin
ALTER DATABASE
[OK]   Set synchronous_commit=local for fastorder_identity_sau_main_dev_db
[INFO] Ensuring pg_hba.conf entry for pgbouncer_admin …
[INFO] Adding pg_hba.conf entries for pgbouncer_admin with cert auth …
[2026-01-02 11:53:14 UTC] USER=unknown EUID=33 PID=2059589 ACTION=-u ARGS=postgres bash
ERROR: Invalid or unauthorized action: -u
[OK]   pg_hba.conf updated and PostgreSQL configuration reloaded
[WARN] pg_hba.conf entry may not have loaded correctly
[INFO] Writing /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini …
[2026-01-02 11:53:15 UTC] USER=www-data EUID=0 PID=2059614 ACTION=fsop ARGS=cp /tmp/tmp.lrdI35SqYr /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
[2026-01-02 11:53:15 UTC] USER=www-data EUID=0 PID=2059623 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
[2026-01-02 11:53:15 UTC] USER=www-data EUID=0 PID=2059632 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
[2026-01-02 11:53:15 UTC] USER=www-data EUID=0 PID=2059641 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbouncer/identity-sau-main-dev /run/pgbouncer/identity-sau-main-dev /var/log/pgbouncer/identity-sau-main-dev
[2026-01-02 11:53:15 UTC] USER=www-data EUID=0 PID=2059650 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[OK]   pgbouncer.ini ready
[INFO] Verifying TLS settings in pgbouncer.ini:
[2026-01-02 11:53:15 UTC] USER=www-data EUID=0 PID=2059660 ACTION=fsop ARGS=grep -E (client_tls_sslmode|server_tls) /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
client_tls_sslmode = verify-full
server_tls_sslmode = verify-full
server_tls_ca_file = /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
server_tls_cert_file = /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
server_tls_key_file  = /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[INFO] Verifying PgBouncer server certificate files:
[2026-01-02 11:53:15 UTC] USER=www-data EUID=0 PID=2059669 ACTION=fsop ARGS=test -r /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[OK]   Server cert readable by postgres: /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-01-02 11:53:15 UTC] USER=www-data EUID=0 PID=2059678 ACTION=fsop ARGS=test -r /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[OK]   Server key readable by postgres: /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[INFO] Verifying coordinator CA certificate:
[2026-01-02 11:53:15 UTC] USER=www-data EUID=0 PID=2059687 ACTION=fsop ARGS=test -r /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[OK]   Coordinator CA readable by postgres: /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] Preflight: stopping any conflicting PgBouncer on 6432 …
[2026-01-02 11:53:15 UTC] USER=www-data EUID=0 PID=2059696 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer.service
[2026-01-02 11:53:15 UTC] USER=www-data EUID=0 PID=2059705 ACTION=passthru ARGS=systemctl stop pgbouncer@identity-sau-main-dev.service
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json?all=1": dial unix /var/run/docker.sock: connect: permission denied
[WARN] Killing existing pgbouncer processes: 1977535
1977630
[2026-01-02 11:53:16 UTC] USER=www-data EUID=0 PID=2059729 ACTION=passthru ARGS=bash -c kill -9 1977535
[2026-01-02 11:53:16 UTC] USER=www-data EUID=0 PID=2059740 ACTION=passthru ARGS=bash -c kill -9 1977630
[2026-01-02 11:53:18 UTC] USER=www-data EUID=0 PID=2059773 ACTION=passthru ARGS=systemctl daemon-reload
[OK]   systemd unit installed: pgbouncer@identity-sau-main-dev.service
[INFO] Running pre-flight IP conflict check for 10.100.1.204:6432 …
[WARN] IP conflict checker not found at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/check-ip-conflicts.sh
[WARN] Skipping pre-flight check - conflicts may occur
[INFO] Starting PgBouncer (identity-sau-main-dev) …
[2026-01-02 11:53:19 UTC] USER=www-data EUID=0 PID=2059868 ACTION=passthru ARGS=systemctl restart pgbouncer@identity-sau-main-dev.service
[2026-01-02 11:53:19 UTC] USER=www-data EUID=0 PID=2059878 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer@identity-sau-main-dev.service
[OK]   Service ACTIVE
[INFO] Verifying auth_file before probing …
[INFO] Auth file contains 4 user(s)
[WARN] Auth file does NOT contain pgbouncer_admin entry - authentication will fail
[INFO] Probing admin console via SSL (psql to database 'pgbouncer') …
[INFO] Retrieved password from vault for admin console probe
[WARN] SSL connection issue detected
[INFO] Attempting connection with sslmode=disable for testing...
[WARN] If this fails, check PgBouncer client_tls_sslmode setting
[WARN] Admin console probe failed (see error below)
psql: error: connection to server at "10.100.1.204", port 6432 failed: SSL error: certificate verify failed
[WARN] Troubleshooting:
[WARN]   1. Check auth_file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[WARN]   2. Test with: PGPASSWORD='kppzNMG6WDrJWGUYcBARr4ME' psql -h 10.100.1.204 -p 6432 -U pgbouncer_admin -d pgbouncer
[WARN]   3. Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@identity-sau-main-dev.service -n 50

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Running Comprehensive PgBouncer Verification Tests
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Password extracted: kppzNMG6WD... (using postgres user certificates)

[INFO] Test 1/7: Admin Console - SHOW POOLS
[WARN] ✗ SHOW POOLS: FAILED
[WARN] Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@identity-sau-main-dev.service -n 50

[INFO] Test 2/7: Admin Console - SHOW VERSION
[WARN] ✗ SHOW VERSION: FAILED

[INFO] Test 3/7: Admin Console - SHOW STATS
[WARN] ✗ SHOW STATS: FAILED

[INFO] Test 4/7: Admin Console - SHOW DATABASES
[WARN] ✗ SHOW DATABASES: FAILED

[INFO] Test 5/7: Admin Console - SHOW CONFIG
[WARN] ✗ SHOW CONFIG: FAILED
psql   "host=db-identity-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_identity_sau_main_dev_db user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME    connect_timeout=5 sslmode=verify-full    sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt    sslcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt    sslkey=/etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key"   --no-psqlrc -Atc 'SELECT version();'

[INFO] Test 6/7: Application Database - SELECT version()
[WARN] ✗ Application database query: FAILED (timeout or connection issue)
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[INFO] Test 7/8: Application Database - Connection Details
[WARN] ✗ Connection details: FAILED (timeout or connection issue)
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[INFO] Test 8/8: End-to-End Application Routing - Pool Verification
[INFO]   Running actual queries through PgBouncer to verify routing and pooling...
[WARN] ✗ End-to-end routing verification: FAILED - All 3 queries failed
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[WARN]    Otherwise check if database fastorder_identity_sau_main_dev_db exists and user pgbouncer_admin has permissions

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Verification Complete - Tests 1-5 PASSED (Admin console verified)
[WARN]   Tests 6-8 FAILED - Application database not accessible
[WARN]   This is expected if Citus is not set up yet
[WARN]   Run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[OK]   PgBouncer is up for identity-sau-main-dev

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Connection Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Password stored in: AWS Secrets Manager (fastorder/db/web/ksa/main/dev/postgresqlidentity/sau/main/dev/coordinator-pgbouncer_admin)
Current password: kppzNMG6WDrJWGUYcBARr4ME

1. Admin Console (using IP address to avoid DNS/SSL issues):
   psql "host=10.100.1.204 port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME sslmode=verify-full sslrootcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

2. Admin Console (using hostname):
   psql "host=db-identity-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME sslmode=verify-full sslrootcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

3. Application Database:
   psql "host=db-identity-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_identity_sau_main_dev_db sslkey=/etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME sslmode=verify-full sslrootcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

4. Using .pgpass file:
   echo "db-identity-sau-main-dev-postgresql-bouncer.fastorder.com:6432:*:pgbouncer_admin:kppzNMG6WDrJWGUYcBARr4ME" >> ~/.pgpass
   chmod 600 ~/.pgpass
   psql -h db-identity-sau-main-dev-postgresql-bouncer.fastorder.com -p 6432 -U pgbouncer_admin -d fastorder_identity_sau_main_dev_db

5. Retrieve password from vault:
   source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
   PGPASSWORD="$(get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password')" \
     psql -h 10.100.1.204 -p 6432 -U pgbouncer_admin -d pgbouncer -c "SHOW POOLS;"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Architecture
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  • Default db 'fastorder_identity_sau_main_dev_db' → Citus coordinator (db-identity-sau-main-dev-postgresql-coordinator.fastorder.com)
  • Worker access: 'fastorder_identity_sau_main_dev_db_worker_1', 'fastorder_identity_sau_main_dev_db_worker_2', … (if exist)
  • Client TLS: require (password auth) / verify-full (mTLS with certs)
  • Server TLS: verify-full (PgBouncer validates PostgreSQL certs)
  • Auth: SCRAM-SHA-256 via /etc/pgbouncer/identity-sau-main-dev/userlist.txt
  • Pool mode: transaction (stateless connections)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Management
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Service Status:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer@identity-sau-main-dev.service
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@identity-sau-main-dev.service

Logs:
  command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@identity-sau-main-dev.service -f
  /usr/local/bin/fastorder-provisioning-wrapper.sh tail -f /var/log/pgbouncer/identity-sau-main-dev/pgbouncer.log

Reload Config:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@identity-sau-main-dev.service

Restart:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@identity-sau-main-dev.service

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Files
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Config:        /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
Auth file:     /etc/pgbouncer/identity-sau-main-dev/userlist.txt
Server cert:   /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.crt
Server key:    /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.key
CA cert:       /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt
PG CA:         /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
Logs:          /var/log/pgbouncer/identity-sau-main-dev/pgbouncer.log

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Troubleshooting
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


If "SASL authentication failed":
  1. Check auth file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/identity-sau-main-dev/userlist.txt
  2. Verify pgbouncer_admin is present with SCRAM hash
  3. Get password from vault:
     source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
     get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password'
  4. Reload PgBouncer: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@identity-sau-main-dev.service

If "no pg_hba.conf entry":
  1. Check pg_hba.conf on coordinator
  2. Add rule: hostssl all pgbouncer_admin 10.100.1.204/32 cert clientcert=verify-full
  3. Reload PostgreSQL

To add users to PgBouncer:
  1. Create user in PostgreSQL with password
  2. Re-run SCRAM dump:
     psql "host=db-identity-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 dbname=postgres user=postgres \
       sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt \
       sslcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt sslkey=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key" \
       -Atc "SELECT '\"' || rolname || '\" \"' || rolpassword || '\"' \
             FROM pg_authid WHERE rolpassword LIKE 'SCRAM-SHA-256%' \
             AND rolcanlogin ORDER BY rolname;" | command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop tee /etc/pgbouncer/identity-sau-main-dev/userlist.txt
  3. Reload: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@identity-sau-main-dev.service

[INFO] Registering PgBouncer node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        identity-sau-main-dev-pgbouncer
[INFO]   Identifier Parent: postgresql
[INFO]   IP:                10.100.1.204
[INFO]   Port:              6432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 426480a5-2f64-4fc0-b2b5-710f9ccb059a
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   PgBouncer node registered to observability API
✓ ✅ PgBouncer setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 03-citus-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] CITUS DISTRIBUTED CLUSTER SETUP
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Phase 1: Installing Citus extension on workers...
[INFO] Phase 2: Setting up coordinator and registering workers...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 📦 PHASE 1: Installing Citus extension on 1 worker(s)...

[INFO] → Worker 1/1: Installing Citus on worker-01...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS CLUSTER SETUP
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔧 Setting up Citus Worker...
[INFO] Temporarily disabling synchronous replication for extension installation...
t
[INFO] Installing Citus extension on worker...
[OK]   Citus extension installed on worker
[INFO] Restoring synchronous replication settings...
t
[INFO] Worker Citus extension installed - registration will happen when coordinator setup runs

[OK]   Citus setup complete for worker-01
[INFO] ═══════════════════════════════════════════════════════════════════════════════
✓   ✅ Citus extension installed on worker-01

✓ ✅ Phase 1 Complete: All 1 workers have Citus extension installed

[INFO] 🔧 PHASE 2: Setting up Citus coordinator and registering workers...

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS CLUSTER SETUP
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔧 Setting up Citus Coordinator...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] DIAGNOSTIC: Configuration Variables
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PG_WORKERS_NUM: 1
[INFO] ENV_ID: identity-sau-main-dev
[INFO] DOMAIN: fastorder.com
[INFO] PORT: 5432
[INFO] SOCKET_DIR: /var/run/postgresql-identity-sau-main-dev-coordinator
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Ensuring postgres client certificates exist for coordinator...
[OK]   Postgres client certificates already exist for coordinator
[INFO] Adding citus_cert_map to coordinator pg_ident.conf...
[OK]   pg_ident.conf updated for coordinator
[INFO] Installing Citus extension on coordinator...
[OK]   Citus extension installed on coordinator (postgres database)
[INFO] Installing Citus extension on application database: fastorder_identity_sau_main_dev_db...
[OK]   Citus extension installed on application database: fastorder_identity_sau_main_dev_db
[INFO] Configuring Citus SSL connection parameters...
[2026-01-02 11:53:35 UTC] USER=www-data EUID=0 PID=2060183 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
[OK]   ✅ Citus SSL connection parameters configured: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[WARN] Node not identified as coordinator, initializing...
[INFO] Checking coordinator configuration...
[INFO] Persisting citus.local_hostname to postgresql.conf...
[2026-01-02 11:53:38 UTC] USER=www-data EUID=0 PID=2060221 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /var/lib/postgresql/17/identity-sau-main-dev/coordinator/postgresql.conf
[2026-01-02 11:53:38 UTC] USER=www-data EUID=0 PID=2060242 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
[OK]   ✅ citus.local_hostname persisted to config and reloaded
[INFO] Configuring coordinator hostname in postgres database: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432

[OK]   ✅ Coordinator hostname set to db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432 in postgres database
[INFO] Checking coordinator configuration in application database: fastorder_identity_sau_main_dev_db...
[WARN] ⚠️  Coordinator registered as 'localhost' in application database, fixing...
[INFO] Configuring coordinator hostname in application database: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432
[OK]   ✅ Coordinator hostname set to db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432 in application database
[INFO] Validating coordinator configuration before worker registration...
[OK]   ✅ Coordinator hostname validated: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[OK]   ✅ citus_tables view is accessible
[INFO] Checking coordinator self-registration...
[OK]   ✅ Coordinator is already self-registered
[INFO] Configuring coordinator shard placement policy...
[OK]   ✅ Coordinator already configured in postgres database (shouldhaveshards = false)
[WARN] ⚠️  Coordinator has 17 shards in fastorder_identity_sau_main_dev_db - cannot set shouldhaveshards=false
[WARN]    You must rebalance shards to workers first, then run this setup again
[WARN]    Skipping shouldhaveshards configuration for application database
[INFO] Registering 1 worker(s) to Citus cluster...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PRE-FLIGHT: Checking worker availability...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking worker worker-01...
[INFO]   FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[OK]   ✅ Worker worker-01 is reachable via SSL
[OK]   All workers are reachable - proceeding with registration

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Adding Citus worker: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Adding citus_cert_map to worker-01 pg_ident.conf...
[OK]   pg_ident.conf updated for worker-01
[INFO] Configuring worker worker-01 HBA for coordinator (10.100.1.213) access...
[OK]   Worker worker-01 HBA configured for coordinator (10.100.1.213)
[INFO] Adding replication rules for 3 standby(s)...
[OK]   Replication rules already exist for worker-01
[INFO] Reloading worker worker-01 to apply HBA changes...
[2026-01-02 11:53:41 UTC] USER=www-data EUID=0 PID=2060366 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
[INFO] Configuring coordinator HBA for worker worker-01 (10.100.1.214) access...
[OK]   Coordinator HBA configured for worker worker-01 (10.100.1.214)
[INFO] Reloading coordinator to apply HBA changes...
[2026-01-02 11:53:42 UTC] USER=www-data EUID=0 PID=2060396 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
[INFO] Ensuring postgres client certificates exist for worker-01...
[OK]   Postgres client certificates already exist for worker-01
[INFO] Configuring citus.node_conninfo on worker-01...
[2026-01-02 11:53:42 UTC] USER=www-data EUID=0 PID=2060412 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
[OK]   citus.node_conninfo configured on worker-01
[INFO] Temporarily relaxing sync-rep on worker worker-01...
t
[OK]   Worker worker-01 sync-rep relaxed (was: sync_commit=on)
[INFO] Ensuring Citus extension on worker databases...
CREATE EXTENSION
CREATE EXTENSION
[INFO] Running citus_add_node with 180s timeout...
NOTICE:  shards are still on the coordinator after adding the new node
HINT:  Use SELECT rebalance_table_shards(); to balance shards data between workers and coordinator or SELECT citus_drain_node('db-identity-sau-main-dev-postgresql-coordinator.fastorder.com',5432); to permanently move shards away from the coordinator.
2
[INFO] Restoring worker worker-01 sync-rep settings...
t
[OK]   Worker worker-01 sync-rep restored
[OK]   ✅ Worker db-identity-sau-main-dev-postgresql-worker-01.fastorder.com successfully added to Citus cluster
[INFO]    Node ID: 2
[INFO]    Registered in: postgres, fastorder_identity_sau_main_dev_db
[OK]   Worker worker-01 registration successful
[INFO] Configuring worker worker-01 shard placement policy...
[OK]   ✅ Worker worker-01 configured to hold shards in all databases


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] POST-REGISTRATION: Verifying cluster state...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Expected workers: 1
[INFO] Registered workers: 1
[OK]   ✅ All 1 workers successfully registered!

[INFO] Citus cluster configuration:
db-identity-sau-main-dev-postgresql-coordinator.fastorder.com  5432  0  t  primary  f
db-identity-sau-main-dev-postgresql-worker-01.fastorder.com    5432  1  t  primary  t

[INFO] Note: groupid=0 is the coordinator, groupid>0 are workers
[INFO]       shouldhaveshards: false=query router only, true=holds data shards

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] FINAL VALIDATION: Verifying configuration persistence...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:53:46 UTC] USER=www-data EUID=0 PID=2060592 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /var/lib/postgresql/17/identity-sau-main-dev/coordinator/postgresql.conf
[OK]   ✅ citus.local_hostname persisted in postgresql.conf
[OK]   ✅ All 1 worker(s) successfully registered and verified

[OK]   ✅ All validation checks passed
[OK]   Citus coordinator setup complete

[OK]   Citus setup complete for coordinator
[INFO] ═══════════════════════════════════════════════════════════════════════════════

✓ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ CITUS CLUSTER SETUP COMPLETED SUCCESSFULLY
✓    Coordinator: Ready and accepting connections
✓    Workers registered: 1
✓ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 05-backup-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up coordinator backup...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for identity-sau-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-01-02 11:53:49 UTC] USER=www-data EUID=0 PID=2060661 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-02 11:53:49 UTC] USER=www-data EUID=0 PID=2060670 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/identity-sau-main-dev
[2026-01-02 11:53:49 UTC] USER=www-data EUID=0 PID=2060679 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-01-02 11:53:49 UTC] USER=www-data EUID=0 PID=2060688 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-02 11:53:49 UTC] USER=www-data EUID=0 PID=2060697 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-02 11:53:49 UTC] USER=www-data EUID=0 PID=2060706 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-02 11:53:57 UTC] USER=www-data EUID=0 PID=2061025 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-02 11:53:57 UTC] USER=www-data EUID=0 PID=2061034 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-02 11:53:57 UTC] USER=www-data EUID=0 PID=2061043 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-01-02 11:53:57 UTC] USER=www-data EUID=0 PID=2061052 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-02 11:53:57 UTC] USER=www-data EUID=0 PID=2061062 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-identity-sau-main-dev
[2026-01-02 11:53:57 UTC] USER=www-data EUID=0 PID=2061084 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-02 11:53:57 UTC] USER=www-data EUID=0 PID=2061095 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-01-02 11:53:57 UTC] USER=www-data EUID=0 PID=2061104 ACTION=fsop ARGS=find /var/lib/postgresql/17/identity-sau-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-01-02 11:53:57 UTC] USER=www-data EUID=0 PID=2061113 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-02 11:53:57 UTC] USER=www-data EUID=0 PID=2061122 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-01-02 11:53:57 UTC] USER=www-data EUID=0 PID=2061131 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-01-02 11:53:57 UTC] USER=www-data EUID=0 PID=2061140 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-02 11:53:57 UTC] USER=www-data EUID=0 PID=2061150 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/PG_VERSION
[2026-01-02 11:53:58 UTC] USER=www-data EUID=0 PID=2061160 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza identity-sau-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-02 11:53:58 UTC] USER=www-data EUID=0 PID=2061216 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-02 11:54:00 UTC] USER=www-data EUID=0 PID=2061238 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 11:54:05 UTC] USER=www-data EUID=0 PID=2061311 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-02 11:54:05 UTC] USER=www-data EUID=0 PID=2061335 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator --log-level-console=info check
2026-01-02 11:54:05.464 P00   INFO: check command begin 2.56.0: --exec-id=2061343-78927c33 --log-level-console=info --log-level-file=debug --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-02 11:54:05.505 P00   INFO: check repo1 configuration (primary)
2026-01-02 11:54:05.523 P00  ERROR: [028]: backup and archive info files exist but do not match the database
                                    HINT: is this the correct stanza?
                                    HINT: did an error occur during stanza-upgrade?
2026-01-02 11:54:05.524 P00   INFO: check command end: aborted with exception [028]
[WARN] ⚠️  Stanza verification failed - this may be normal if WAL archiving hasn't started yet
[WARN]    The backup system is configured and will work once WAL segments are generated

[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-02 11:54:05 UTC] USER=www-data EUID=0 PID=2061356 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-02 11:54:05 UTC] USER=www-data EUID=0 PID=2061365 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-02 11:54:05 UTC] USER=www-data EUID=0 PID=2061383 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[2026-01-02 11:54:05 UTC] USER=www-data EUID=0 PID=2061392 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-02 11:54:05 UTC] USER=www-data EUID=0 PID=2061410 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-identity-sau-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-01-02 11:54:05 UTC] USER=www-data EUID=0 PID=2061428 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 11:54:05 UTC] USER=www-data EUID=0 PID=2061437 ACTION=fsop ARGS=sed -i s|__ENV_ID__|identity-sau-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 11:54:05 UTC] USER=www-data EUID=0 PID=2061446 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/var/lib/postgresql/17/identity-sau-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 11:54:06 UTC] USER=www-data EUID=0 PID=2061455 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 11:54:06 UTC] USER=www-data EUID=0 PID=2061464 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-02 11:54:06.211 P00   INFO: start command begin 2.56.0: --exec-id=2061485-64989c34 --log-level-console=info --log-level-file=debug --stanza=identity-sau-main-dev-coordinator
2026-01-02 11:54:06.212 P00   WARN: stop file does not exist for stanza identity-sau-main-dev-coordinator
2026-01-02 11:54:06.212 P00   INFO: start command end: completed successfully (6ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-02 11:54:06.289 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=2061496-8a39f6fd --log-level-console=info --log-level-file=debug --no-online --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-02 11:54:06.291 P00   INFO: stanza-upgrade for stanza 'identity-sau-main-dev-coordinator' on repo1
2026-01-02 11:54:06.314 P00   INFO: stanza-upgrade command end: completed successfully (34ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-02 11:54:06 UTC] USER=www-data EUID=0 PID=2061500 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260102-115406.log
[2026-01-02 11:54:06 UTC] USER=www-data EUID=0 PID=2061509 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260102-115406.log
[2026-01-02 11:54:06 UTC] USER=www-data EUID=0 PID=2061518 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260102-115406.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-01-02 11:54:17 UTC] USER=www-data EUID=0 PID=2061609 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-2060633.log /var/log/pgbackrest/initial-backup-20260102-115406.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260102-115406.log
   2026-01-02 11:54:16.941 P00   INFO: repo1: remove expired backup 20260102-085913F
   2026-01-02 11:54:17.026 P00   INFO: repo1: 17-25 remove archive, start = 000000010000000000000004, stop = 000000010000000000000006
   2026-01-02 11:54:17.027 P00   INFO: repo1: 17-26 no archive to remove
   2026-01-02 11:54:17.027 P00   INFO: repo1: 17-27 remove archive, start = 000000010000000000000003, stop = 000000010000000000000003
   2026-01-02 11:54:17.027 P00   INFO: expire command end: completed successfully (110ms)

[INFO] Current backups:
stanza: identity-sau-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000007/000000010000000000000018

        full backup: 20260102-085944F
            timestamp start/stop: 2026-01-02 08:59:44+00 / 2026-01-02 08:59:49+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.3MB, database backup size: 37.3MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

    db (prior)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000012

        full backup: 20260102-110513F
            timestamp start/stop: 2026-01-02 11:05:13+00 / 2026-01-02 11:05:24+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.3MB, database backup size: 37.3MB
            repo1: backup set size: 5.8MB, backup size: 5.8MB

        full backup: 20260102-110544F
            timestamp start/stop: 2026-01-02 11:05:44+00 / 2026-01-02 11:05:48+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.3MB, database backup size: 37.3MB
            repo1: backup set size: 5.8MB, backup size: 5.8MB

    db (current)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000004

        full backup: 20260102-115406F
            timestamp start/stop: 2026-01-02 11:54:06+00 / 2026-01-02 11:54:16+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.4MB, database backup size: 37.4MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         identity-sau-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/identity-sau-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up worker backups for 1 worker(s)...
[INFO] Setting up backup for: worker-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for identity-sau-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-01-02 11:54:17 UTC] USER=www-data EUID=0 PID=2061667 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-02 11:54:17 UTC] USER=www-data EUID=0 PID=2061676 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/identity-sau-main-dev
[2026-01-02 11:54:17 UTC] USER=www-data EUID=0 PID=2061685 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-01-02 11:54:17 UTC] USER=www-data EUID=0 PID=2061694 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-02 11:54:17 UTC] USER=www-data EUID=0 PID=2061703 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-02 11:54:17 UTC] USER=www-data EUID=0 PID=2061712 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-02 11:54:25 UTC] USER=www-data EUID=0 PID=2061770 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-02 11:54:26 UTC] USER=www-data EUID=0 PID=2061779 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-02 11:54:26 UTC] USER=www-data EUID=0 PID=2061788 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-01-02 11:54:26 UTC] USER=www-data EUID=0 PID=2061797 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-02 11:54:26 UTC] USER=www-data EUID=0 PID=2061806 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-identity-sau-main-dev
[2026-01-02 11:54:26 UTC] USER=www-data EUID=0 PID=2061827 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-02 11:54:26 UTC] USER=www-data EUID=0 PID=2061836 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-01-02 11:54:26 UTC] USER=www-data EUID=0 PID=2061845 ACTION=fsop ARGS=find /var/lib/postgresql/17/identity-sau-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-01-02 11:54:26 UTC] USER=www-data EUID=0 PID=2061854 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-02 11:54:26 UTC] USER=www-data EUID=0 PID=2061863 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-01-02 11:54:26 UTC] USER=www-data EUID=0 PID=2061874 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-01-02 11:54:26 UTC] USER=www-data EUID=0 PID=2061883 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-02 11:54:26 UTC] USER=www-data EUID=0 PID=2061892 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/PG_VERSION
[2026-01-02 11:54:26 UTC] USER=www-data EUID=0 PID=2061902 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza identity-sau-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-02 11:54:28 UTC] USER=www-data EUID=0 PID=2061971 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-02 11:54:30 UTC] USER=www-data EUID=0 PID=2061992 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 11:54:34 UTC] USER=www-data EUID=0 PID=2062040 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-02 11:54:34 UTC] USER=www-data EUID=0 PID=2062064 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator --log-level-console=info check
2026-01-02 11:54:34.501 P00   INFO: check command begin 2.56.0: --exec-id=2062071-7d5c683d --log-level-console=info --log-level-file=debug --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-02 11:54:34.551 P00   INFO: check repo1 configuration (primary)
2026-01-02 11:54:34.618 P00   INFO: check repo1 archive for WAL (primary)
2026-01-02 11:54:34.919 P00   INFO: WAL segment 000000010000000000000006 successfully archived to '/var/lib/pgbackrest/backup/identity-sau-main-dev/archive/identity-sau-main-dev-coordinator/17-27/0000000100000000/000000010000000000000006-8cb49475d6191c00392ed6aa22789acc5d9b8243.lz4' on repo1
2026-01-02 11:54:34.919 P00   INFO: check command end: completed successfully (423ms)
[INFO] ✅ Stanza verification passed

[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-02 11:54:34 UTC] USER=www-data EUID=0 PID=2062096 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-02 11:54:35 UTC] USER=www-data EUID=0 PID=2062105 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-02 11:54:35 UTC] USER=www-data EUID=0 PID=2062123 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[2026-01-02 11:54:35 UTC] USER=www-data EUID=0 PID=2062132 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-02 11:54:35 UTC] USER=www-data EUID=0 PID=2062150 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-identity-sau-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-01-02 11:54:35 UTC] USER=www-data EUID=0 PID=2062168 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 11:54:35 UTC] USER=www-data EUID=0 PID=2062177 ACTION=fsop ARGS=sed -i s|__ENV_ID__|identity-sau-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 11:54:35 UTC] USER=www-data EUID=0 PID=2062186 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/var/lib/postgresql/17/identity-sau-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 11:54:35 UTC] USER=www-data EUID=0 PID=2062195 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 11:54:35 UTC] USER=www-data EUID=0 PID=2062204 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-02 11:54:35.559 P00   INFO: start command begin 2.56.0: --exec-id=2062227-577aad3f --log-level-console=info --log-level-file=debug --stanza=identity-sau-main-dev-coordinator
2026-01-02 11:54:35.560 P00   WARN: stop file does not exist for stanza identity-sau-main-dev-coordinator
2026-01-02 11:54:35.560 P00   INFO: start command end: completed successfully (6ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-02 11:54:35.627 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=2062238-63ed526d --log-level-console=info --log-level-file=debug --no-online --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-02 11:54:35.629 P00   INFO: stanza-upgrade for stanza 'identity-sau-main-dev-coordinator' on repo1
2026-01-02 11:54:35.632 P00   INFO: stanza 'identity-sau-main-dev-coordinator' on repo1 is already up to date
2026-01-02 11:54:35.633 P00   INFO: stanza-upgrade command end: completed successfully (14ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-02 11:54:35 UTC] USER=www-data EUID=0 PID=2062242 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260102-115435.log
[2026-01-02 11:54:35 UTC] USER=www-data EUID=0 PID=2062251 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260102-115435.log
[2026-01-02 11:54:35 UTC] USER=www-data EUID=0 PID=2062260 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260102-115435.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-01-02 11:54:40 UTC] USER=www-data EUID=0 PID=2062321 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-2061634.log /var/log/pgbackrest/initial-backup-20260102-115435.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260102-115435.log
   2026-01-02 11:54:40.172 P00   INFO: repo1: remove expired backup 20260102-085944F
   2026-01-02 11:54:40.254 P00   INFO: repo1: remove archive path /var/lib/pgbackrest/backup/identity-sau-main-dev/archive/identity-sau-main-dev-coordinator/17-25
   2026-01-02 11:54:40.263 P00   INFO: repo1: 17-26 no archive to remove
   2026-01-02 11:54:40.263 P00   INFO: repo1: 17-27 no archive to remove
   2026-01-02 11:54:40.264 P00   INFO: expire command end: completed successfully (110ms)

[INFO] Current backups:
stanza: identity-sau-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000012

        full backup: 20260102-110513F
            timestamp start/stop: 2026-01-02 11:05:13+00 / 2026-01-02 11:05:24+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.3MB, database backup size: 37.3MB
            repo1: backup set size: 5.8MB, backup size: 5.8MB

        full backup: 20260102-110544F
            timestamp start/stop: 2026-01-02 11:05:44+00 / 2026-01-02 11:05:48+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.3MB, database backup size: 37.3MB
            repo1: backup set size: 5.8MB, backup size: 5.8MB

    db (current)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000007

        full backup: 20260102-115406F
            timestamp start/stop: 2026-01-02 11:54:06+00 / 2026-01-02 11:54:16+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.4MB, database backup size: 37.4MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

        full backup: 20260102-115435F
            timestamp start/stop: 2026-01-02 11:54:35+00 / 2026-01-02 11:54:39+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.4MB, database backup size: 37.4MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         identity-sau-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/identity-sau-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ Backup setup completed for coordinator and all workers

[INFO] Skipping 06-distribute-tables-canary.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 07-distribute-tables.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:54:42 UTC] USER=unknown EUID=33 PID=2062376 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 11:54:42 UTC] USER=unknown EUID=33 PID=2062383 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 11:54:42 UTC] USER=unknown EUID=33 PID=2062390 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 11:54:42 UTC] USER=unknown EUID=33 PID=2062397 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS TABLE DISTRIBUTION
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔐 Secure connection established
[INFO]    Host: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432
[INFO]    Database: fastorder_identity_sau_main_dev_db
[INFO]    SSL: verify-full (TLS 1.2+)
[INFO]    Timeouts: statement=120s, idle_tx=300s

[INFO] 🔍 Running preflight checks...
[INFO] Testing database connectivity...
[OK]   ✅ Database connection successful
[OK]   ✅ Connected to correct database: fastorder_identity_sau_main_dev_db
[INFO] Checking Citus extension in database fastorder_identity_sau_main_dev_db...
[OK]   Citus version: 13.2-1
[INFO] Checking worker registration...
[OK]   Registered workers: 1
[INFO] Worker nodes:
[INFO]                             nodename                           | nodeport | isactive | noderole 
[INFO]   -------------------------------------------------------------+----------+----------+----------
[INFO]    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com |     5432 | t        | primary
[INFO]   (1 row)
[INFO]   

[INFO] 📊 Starting table distribution...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Distributing: auth.login_account
[INFO] Description: User authentication table - distributed by region for tenant isolation
[INFO] Shard key: region_hint
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📝 Current rows: 0
[INFO] Checking constraints compatibility with Citus...
[OK]   ✅ No conflicting constraints found
[OK]   ✅ Table already distributed - skipping
[INFO]    Distribution column: region_hint
[OK]   ✅ Data integrity verified (0 rows)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[OK]   ✅ All tables distributed successfully!
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 📊 Citus Cluster Summary:

[INFO] Distributed tables:
[INFO]            table          |   type    | shard_key | shards | size  
[INFO]   ------------------------+-----------+-----------+--------+-------
[INFO]    core.tenant            | reference | <none>    |      1 | 24 kB
[INFO]    core.realm             | local     | <none>    |      1 | 40 kB
[INFO]    core.identity          | local     | <none>    |      1 | 72 kB
[INFO]    core.device            | local     | <none>    |      1 | 48 kB
[INFO]    core.identity_account  | local     | <none>    |      1 | 48 kB
[INFO]    core.identity_mfa      | local     | <none>    |      1 | 40 kB
[INFO]    core.external_idp_link | local     | <none>    |      1 | 48 kB
[INFO]    policy.client          | local     | <none>    |      1 | 56 kB
[INFO]    policy.resource        | local     | <none>    |      1 | 48 kB
[INFO]    policy.scope           | local     | <none>    |      1 | 40 kB
[INFO]    policy.permission      | local     | <none>    |      1 | 48 kB
[INFO]    policy.role            | local     | <none>    |      1 | 56 kB
[INFO]    policy.role_permission | local     | <none>    |      1 | 24 kB
[INFO]    policy.identity_role   | local     | <none>    |      1 | 40 kB
[INFO]    policy.policy_rule     | local     | <none>    |      1 | 48 kB
[INFO]    policy.api_key         | local     | <none>    |      1 | 56 kB
[INFO]    auth.login_account     | reference | <none>    |      1 | 48 kB
[INFO]   (17 rows)
[INFO]   

[INFO] Worker capacity:
[INFO]    worker | total_shards | total_size 
[INFO]   --------+--------------+------------
[INFO]   (0 rows)
[INFO]   

[OK]   Citus table distribution complete

[INFO] Skipping 08-distribute-tables-rollback.sh (rollback script - run manually only)
[INFO] Skipping 09-distribute-tables-test.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 10-setup-cdc.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] CDC PIPELINE SETUP (Debezium + Elasticsearch Sink)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Log file: /var/log/fastorder/cdc/10-setup-cdc-*.log

[INFO] Running CDC setup for identifier: coordinator
[2026-01-02 11:54:53] ==========================================
[2026-01-02 11:54:53] CDC SETUP SCRIPT STARTED
[2026-01-02 11:54:53] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260102_115453.log
[2026-01-02 11:54:53] ==========================================
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 11:54:53] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:54:53]   CDC Pipeline Setup (Debezium + ES Sink)
[2026-01-02 11:54:53] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:54:54]   Environment: identity-sau-main-dev
[2026-01-02 11:54:54]   Identifier:  coordinator
[2026-01-02 11:54:54]   Service:     identity
[2026-01-02 11:54:54] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:54:54] 📂 CDC_BASE_DIR exists: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc
[2026-01-02 11:54:54] Looking for service folder: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity
[2026-01-02 11:54:54] 
[2026-01-02 11:54:54] 📂 Found CDC configuration for service: identity
[2026-01-02 11:54:54] Scanning for subservice directories in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity
[2026-01-02 11:54:54] Found subservice: login, checking for steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps
[2026-01-02 11:54:54] 
[2026-01-02 11:54:54] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:54:54]   Setting up CDC for: identity/login
[2026-01-02 11:54:54] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:54:54] Found 7 step script(s) in /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps
[2026-01-02 11:54:54] 
[2026-01-02 11:54:54] 🔧 Running: 01-setup-debezium-auth-login.sh
[2026-01-02 11:54:54]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/01-setup-debezium-auth-login.sh
[2026-01-02 11:54:54]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Debezium CDC Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔍 Verifying Kafka infrastructure...
✅ db-identity-sau-main-dev-postgresql.fastorder.com resolves to 10.100.1.213
🔐 psql will use client cert for mTLS.
🔐 Retrieving credentials from secrets vault...
   Clearing cached credentials for coordinator...
✅ Credentials retrieved from secrets vault
🔐 Syncing debezium_user password in PostgreSQL...
✅ debezium_user password synchronized
🔍 Checking PostgreSQL SSL status...
✅ Server SSL is ON (verify-full + client cert).
🔧 Applying publication & grants over TLS…
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

NOTICE:  publication "cdc_pub_identity" does not exist, skipping
DROP PUBLICATION
CREATE PUBLICATION
SET
NOTICE:  Added shard table auth.login_account_102024 to publication
DO
RESET
GRANT
GRANT
GRANT
✅ Publication & grants done (including Citus shard table).
⏳ Waiting for Kafka Connect @ https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors…
[2026-01-02 11:55:00] 🔗 Waiting for Kafka Connect at: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
[2026-01-02 11:55:00] ⏳ Waiting for HTTP endpoint: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
[2026-01-02 11:55:00]    Expected codes: 200,500, timeout: 300s
[2026-01-02 11:55:00] ✅ HTTP endpoint ready: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083 (code: 200, took: 0s)
[2026-01-02 11:55:00] 🔄 Testing Connect worker readiness...
[2026-01-02 11:55:00] ✅ Kafka Connect worker ready
🧹 Cleaning up existing Debezium connector and slot (if any)...
   Step 0a: Also resetting ES Sink connector offsets (required for coordinated reset)...
   → Stopping ES Sink connector pg_identity_sau_main_dev_coordinator_es_sink...
   → Deleting ES Sink connector offsets...
   ✓ ES Sink offsets deleted successfully (HTTP 200)
   → Deleting ES Sink connector (will be recreated by 02-setup-es-sink.sh)...
   ✓ ES Sink connector cleanup complete
   Step 0b: Clearing stale Debezium connector offsets from Kafka Connect...
   → Stopping connector pg_identity_sau_main_dev_debezium_postgres...
   → Deleting connector offsets (forces fresh snapshot)...
   ✓ Connector offsets deleted successfully (HTTP 200)
   Step 1: Ensuring connector is completely removed...
   Deleting connector: pg_identity_sau_main_dev_debezium_postgres (attempt 1/10)
   ✓ Connector pg_identity_sau_main_dev_debezium_postgres does not exist (HTTP 404)
   Step 2: Waiting for replication slot to become inactive...
   ✓ Slot slot_identity_sau_main_dev does not exist (clean state)
   Step 3: Dropping replication slot...
   ✓ Slot slot_identity_sau_main_dev already dropped
   Step 4: Final verification...
✅ Cleanup complete - environment is clean for fresh CDC snapshot
🔐 Checking Debezium SSL certificate permissions...
🔍 Validating Debezium SSL certificates...
🔐 Connector will use mTLS to Postgres.
  ✓ Certificate: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt
  ✓ Key: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key
  ✓ Root CA: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
ℹ️  Skipping pre-flight connectivity test (will be validated by Kafka Connect)
📤 Upserting connector: PUT https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_identity_sau_main_dev_debezium_postgres/config
   Attempt 1/5: Sending PUT request to Kafka Connect...
   (This may take up to 60s as Connect validates the configuration)
   ✅ Success (HTTP 201)

🌐 HTTP Response: 201
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Response body:
{
  "name": "pg_identity_sau_main_dev_debezium_postgres",
  "config": {
    "name": "pg_identity_sau_main_dev_debezium_postgres",
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "plugin.name": "pgoutput",
    "database.hostname": "db-identity-sau-main-dev-postgresql.fastorder.com",
    "database.port": "5432",
    "database.dbname": "fastorder_identity_sau_main_dev_db",
    "database.user": "debezium_user",
    "database.password": "p6uvPVcDhG0o2EA8L0ESe0AC6",
    "database.sslmode": "verify-full",
    "database.sslrootcert": "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt",
    "database.sslcert": "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt",
    "database.sslkey": "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key",
    "publication.name": "cdc_pub_identity",
    "publication.autocreate.mode": "disabled",
    "slot.name": "slot_identity_sau_main_dev",
    "topic.prefix": "identity_sau_main_dev_cdc",
    "schema.include.list": "auth",
    "table.include.list": "auth.login_account,auth.login_account_[0-9]+",
    "transforms": "unwrap,route",
    "transforms.unwrap.add.fields": "op,ts_ms",
    "transforms.unwrap.delete.handling.mode": "rewrite",
    "transforms.unwrap.drop.tombstones": "false",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
    "transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.route.regex": "^identity_sau_main_dev_cdc\\.auth\\.login_account(_[0-9]+)?$",
    "transforms.route.replacement": "identity_sau_main_dev_account_router",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",
    "snapshot.mode": "always"
  },
  "tasks": [],
  "type": "source"
}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Connector upserted.
🔄 Verifying connector task startup...
✅ Debezium connector task is RUNNING
ℹ️  Source table auth.login_account has 0 rows.
ℹ️  Snapshot will be metadata-only; offsets may stay empty until first change.
⏳ Waiting for Debezium initial snapshot to complete...
   📊 Slot status: restart_lsn=0/8016028, confirmed_flush_lsn=0/8016060
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (0s elapsed)
   ⏳ Snapshot in progress... (5s elapsed)
   ⏳ Snapshot in progress... (10s elapsed)
   📊 Slot status: restart_lsn=0/8016028, confirmed_flush_lsn=0/8016060
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (15s elapsed)
   ⏳ Snapshot in progress... (20s elapsed)
   ⏳ Snapshot in progress... (25s elapsed)
   📊 Slot status: restart_lsn=0/8016028, confirmed_flush_lsn=0/8016060
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (30s elapsed)
   ⏳ Snapshot in progress... (35s elapsed)
   ⏳ Snapshot in progress... (40s elapsed)
   📊 Slot status: restart_lsn=0/8016028, confirmed_flush_lsn=0/8016060
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (45s elapsed)
   ⏳ Snapshot in progress... (50s elapsed)
   ⏳ Snapshot in progress... (55s elapsed)
   📊 Slot status: restart_lsn=0/8016028, confirmed_flush_lsn=0/8016060
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (60s elapsed)
   ⏳ Snapshot in progress... (65s elapsed)
   ⏳ Snapshot in progress... (70s elapsed)
   📊 Slot status: restart_lsn=0/8016028, confirmed_flush_lsn=0/8016060
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (75s elapsed)
   ⏳ Snapshot in progress... (80s elapsed)
   ⏳ Snapshot in progress... (85s elapsed)
   📊 Slot status: restart_lsn=0/8016028, confirmed_flush_lsn=0/8016060
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (90s elapsed)
   ⏳ Snapshot in progress... (95s elapsed)
   ⏳ Snapshot in progress... (100s elapsed)
   📊 Slot status: restart_lsn=0/8016028, confirmed_flush_lsn=0/8016060
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (105s elapsed)
   ⏳ Snapshot in progress... (110s elapsed)
   ⏳ Snapshot in progress... (115s elapsed)

⚠️  WARNING: Snapshot wait timeout (120s) on EMPTY table.
   Offsets are still empty, but source table has 0 rows.
   Proceeding anyway – CDC health will be verified by test inserts.

✅ Debezium connector is RUNNING after snapshot
🔍 Final verification: Checking Debezium offsets are recorded...
   ℹ️  Source table auth.login_account has 0 rows
   ℹ️  Skipping offset verification (no data to snapshot)
✅ Debezium connector verified RUNNING (empty source table)
🔄 Phase 2: Updating connector to snapshot.mode=initial...
✅ Connector updated to snapshot.mode=initial (HTTP 200)
✅ Connector verified RUNNING after Phase 2 update
✅ Debezium connector configured successfully (two-phase snapshot complete)
[2026-01-02 11:57:33] ✅ Completed: 01-setup-debezium-auth-login.sh
[2026-01-02 11:57:33] 
[2026-01-02 11:57:33] 🔧 Running: 02-setup-es-sink.sh
[2026-01-02 11:57:33]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/02-setup-es-sink.sh
[2026-01-02 11:57:33]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[WARN] Master/coordinator not found, using node-01
[INFO] Using ES domain: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com
🔐 Retrieving keystore passwords from secrets manager...
[INFO] Retrieving Kafka truststore password...
✅ Retrieved passwords from remote backend
✅ Retrieved Kafka truststore password
[INFO] Retrieving Elasticsearch P12 password...
[INFO] 🔍 Checking secrets backend (provider: aws)...
✅ Retrieved passwords from remote backend
[INFO] ✅ Using existing passwords from backend
✅ Retrieved/generated Elasticsearch P12 password
✅ Keystore passwords retrieved successfully
   - Kafka truststore password: yOb0eqkA... (32 chars)
   - ES P12 password: 8siDJx7z... (32 chars)
[INFO] 🔐 Clearing cached ES credentials to ensure fresh retrieval...
[INFO] [INFO] ✅ Using ES password from centralized secrets vault (identifier: node-01)
[INFO] 🔐 Verifying Elasticsearch accepts client certificate...
[INFO] ✅ Elasticsearch accepting client certificate
[INFO] 🔐 Setting up ES client keystore using Kafka client certificate...
[INFO]    Certificate: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem (signed by Fastorder RA Root CA)
[INFO] 📋 Creating ES client P12 keystore from Kafka client certificate...
[2026-01-02 11:57:45 UTC] USER=www-data EUID=0 PID=2065591 ACTION=fsop ARGS=mv /tmp/es-client-2065385.p12 /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[2026-01-02 11:57:45 UTC] USER=www-data EUID=0 PID=2065601 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[2026-01-02 11:57:45 UTC] USER=www-data EUID=0 PID=2065610 ACTION=fsop ARGS=chmod 600 /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[INFO] ✅ Created ES client keystore: /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[INFO]    Using Kafka client cert signed by Fastorder RA Root CA
[INFO] ℹ️  Using Kafka truststore and adding ES CA certificate
[2026-01-02 11:57:45 UTC] USER=www-data EUID=0 PID=2065619 ACTION=fsop ARGS=test -f /opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks
[INFO] ✅ ES CA already in truststore
[INFO] [INFO] 🔗 Waiting for Kafka Connect at: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
[INFO] [INFO] ✅ Connect HTTP ready (code 200)
[INFO] [INFO] 🔍 Verifying Debezium connector snapshot status...
[INFO] [INFO] ℹ️  Source table auth.login_account has 0 rows.
[INFO] [INFO]    Skipping Debezium snapshot wait (metadata-only snapshot on empty table).
[INFO] [INFO] 🔌 Cleaning up existing ES Sink connector: pg_identity_sau_main_dev_coordinator_es_sink
[INFO] [INFO]    → Deleting connector...
[INFO] [INFO]    HTTP 404 (404 is fine)
[INFO] [INFO] 🔐 Validating Elasticsearch credentials...
[INFO] [INFO] ✅ ES credentials validated successfully
[INFO] [INFO] 🔧 Creating required Elasticsearch ingest pipelines: identity-embed-pipeline-001
[INFO] [INFO] ✅ Pipeline identity-embed-pipeline-001 created successfully
[INFO] [INFO] 🔧 Ensuring CDC index has no default_pipeline requirement...
[INFO] [INFO] ✅ Removed default_pipeline from index (if any)
[INFO] [INFO] 🔧 Ensuring dynamic mapping is enabled...
[INFO] [INFO] ✅ Dynamic mapping enabled for identity_sau_main_dev_account_router
[DEBUG] ES_TRUSTSTORE=/opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks
[DEBUG] ES_CLIENT_P12=/opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[DEBUG] TRUSTSTORE_PASS=yOb0eqkA...
[DEBUG] P12_PASS=8siDJx7z...
== Outgoing connector config (snippet) ==
2:  "name": "pg_identity_sau_main_dev_coordinator_es_sink",
6:  "connection.url": "https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200",
19:  "index": "identity_sau_main_dev_account_router",
[INFO] ⚠️  Skipping pre-validation - will validate on PUT...
[INFO] [INFO] ✅ Proceeding to PUT
[2026-01-02 11:57:47] [1/3] Upserting connector via PUT https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_identity_sau_main_dev_coordinator_es_sink/config
🌐 HTTP 201
✅ Connector created/updated successfully
{
  "name": "pg_identity_sau_main_dev_coordinator_es_sink",
  "config": {
    "name": "pg_identity_sau_main_dev_coordinator_es_sink",
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "tasks.max": "1",
    "topics": "identity_sau_main_dev_account_router",
    "connection.url": "https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200",
    "elastic.security.protocol": "SSL",
    "elastic.https.ssl.hostname.verification": "true",
    "elastic.https.ssl.truststore.location": "/opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks",
    "elastic.https.ssl.truststore.password": "yOb0eqkAqtj8HEWebgA7nf04YlqsLw44",
    "elastic.https.ssl.truststore.type": "JKS",
    "elastic.https.ssl.keystore.location": "/opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12",
    "elastic.https.ssl.keystore.password": "8siDJx7zdDhhu5iMMZwnhZfTaGFSgCvh",
    "elastic.https.ssl.keystore.type": "PKCS12",
    "elastic.username": "elastic",
    "elastic.password": "T+kMy0e84aGeV204NzYK",
    "connection.username": "elastic",
    "connection.password": "T+kMy0e84aGeV204NzYK",
    "index": "identity_sau_main_dev_account_router",
    "key.ignore": "true",
    "schema.ignore": "true",
    "behavior.on.null.values": "delete",
    "write.method": "upsert",
    "type.name": "_doc",
    "max.in.flight.requests": "1",
    "batch.size": "2000",
    "linger.ms": "100",
    "flush.timeout.ms": "60000",
    "max.retries": "10",
    "retry.backoff.ms": "5000",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false"
  },
  "tasks": [],
  "type": "sink"
}
{
  "pg_identity_sau_main_dev_debezium_postgres": {
    "status": {
      "name": "pg_identity_sau_main_dev_debezium_postgres",
      "connector": {
        "state": "RUNNING",
        "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
      },
      "tasks": [
        {
          "id": 0,
          "state": "RUNNING",
          "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
        }
      ],
      "type": "source"
    }
  },
  "pg_identity_sau_to_universe_main_dev_es_sink": {
    "status": {
      "name": "pg_identity_sau_to_universe_main_dev_es_sink",
      "connector": {
        "state": "RUNNING",
        "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
      },
      "tasks": [
        {
          "id": 0,
          "state": "RUNNING",
          "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
        }
      ],
      "type": "sink"
    }
  },
  "pg_identity_sau_main_dev_coordinator_es_sink": {
    "status": {
      "name": "pg_identity_sau_main_dev_coordinator_es_sink",
      "connector": {
        "state": "RUNNING",
        "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
      },
      "tasks": [],
      "type": "sink"
    }
  }
}
[INFO] [INFO] 🔗 Creating ES alias for application compatibility...
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (0s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (5s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (10s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (15s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (20s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (25s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (30s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (35s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (40s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (45s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (50s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (55s)
[WARN] ⚠️  ES index not created within 60s, skipping alias creation

🔍 Final verification: Checking ES document count...
   PostgreSQL auth.login_account: 0 rows
ℹ️  PostgreSQL table is empty - skipping ES verification
✅ Done.
[2026-01-02 11:58:49] ✅ Completed: 02-setup-es-sink.sh
[2026-01-02 11:58:49] 
[2026-01-02 11:58:49] 🔧 Running: 03-setup-es-universe-sink.sh
[2026-01-02 11:58:49]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/03-setup-es-universe-sink.sh
[2026-01-02 11:58:49]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Universe Identity ES Sink Setup (Dual-Sink Pattern)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Source Zone:  sau
  Connector:      pg_identity_sau_to_universe_main_dev_es_sink
  Source Topic:   identity_sau_main_dev_account_router
  Universe ES:      search-identity-universe-main-dev.fastorder.com:9200
  Universe Index:   identity_universe_main_dev_account_router
  Zone Field:   zone: "sau" (added to each document)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Retrieving keystore passwords from secrets manager...
✅ Retrieved passwords from remote backend
✅ Retrieved Kafka truststore password
[INFO] 🔍 Checking secrets backend (provider: aws)...
✅ Retrieved passwords from remote backend
[INFO] ✅ Using existing passwords from backend
✅ Retrieved/generated Elasticsearch P12 password
🔐 Retrieving Universe ES password...
[INFO] [INFO] ✅ Retrieved Universe ES password from vault (identifier: node-01)
[INFO] 🔐 Setting up ES client keystore using Kafka client certificate...
[INFO]    Certificate: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem (signed by Fastorder RA Root CA)
[2026-01-02 11:58:59 UTC] USER=www-data EUID=0 PID=2067061 ACTION=fsop ARGS=mv /tmp/es-client-2066577.p12 /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[2026-01-02 11:58:59 UTC] USER=www-data EUID=0 PID=2067070 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[2026-01-02 11:58:59 UTC] USER=www-data EUID=0 PID=2067079 ACTION=fsop ARGS=chmod 600 /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[INFO] ✅ Created ES client keystore: /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[INFO] 🔐 Checking Universe ES CA in truststore...
[INFO] ✅ Universe ES CA already in truststore
[INFO] 🔄 Restarting Kafka Connect to load updated truststore...
[2026-01-02 11:59:01 UTC] USER=www-data EUID=0 PID=2067166 ACTION=passthru ARGS=systemctl restart confluent-connect-identity-sau-main-dev_coordinator.service
[INFO] ✅ Kafka Connect restarted: confluent-connect-identity-sau-main-dev_coordinator.service
[INFO]    Waiting for Kafka Connect to be ready (up to 120s)...
[INFO] ✅ Kafka Connect is ready after 9x2 seconds
[INFO] [INFO] ✅ Using unified PKI (Kafka client cert) for Universe ES mTLS
🔍 Verifying Universe ES connectivity...
✅ Universe ES cluster is reachable
   Cluster: fastorder-identity-universe-main-dev, Status: green
👤 Ensuring app_user exists on Universe ES...
[2026-01-02 11:59:20 UTC] USER=www-data EUID=0 PID=2067751 ACTION=passthru ARGS=grep -q ^app_user: /etc/elasticsearch/identity-universe-main-dev/node-01/users
✅ app_user already exists on Universe ES
🔧 Creating Universe ES index if not exists...
✅ Universe index already exists: identity_universe_main_dev_account_router
[INFO] 🔧 Ensuring index settings are compatible with CDC...
[INFO]    → Removing default_pipeline from index...
[INFO]    → Enabling dynamic mapping...
[INFO] ✅ Index settings updated for CDC compatibility
[INFO] [INFO] 🔗 Waiting for Kafka Connect at: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
[INFO] [INFO] ✅ Connect HTTP ready (code 200)
[INFO] [INFO] 🔌 Cleaning up existing Universe ES Sink connector: pg_identity_sau_to_universe_main_dev_es_sink
[INFO] [INFO]    → Stopping connector pg_identity_sau_to_universe_main_dev_es_sink...
[INFO] [INFO]    → Deleting connector offsets...
[INFO] [INFO]    → Deleting connector...

📤 Creating Universe Identity ES Sink connector...
   PUT https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_identity_sau_to_universe_main_dev_es_sink/config

   Transform: zone = "sau" (added to every document)
[1/3] Upserting connector via PUT https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_identity_sau_to_universe_main_dev_es_sink/config
🌐 HTTP 201
✅ Connector created/updated successfully
{
  "name": "pg_identity_sau_to_universe_main_dev_es_sink",
  "config": {
    "name": "pg_identity_sau_to_universe_main_dev_es_sink",
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "tasks.max": "1",
    "topics": "identity_sau_main_dev_account_router",
    "connection.url": "https://search-identity-universe-main-dev.fastorder.com:9200",
    "connection.username": "elastic",
    "connection.password": "h+yhmN0YCeA_Vt7epnWZ",
    "elastic.security.protocol": "SSL",
    "elastic.https.ssl.truststore.location": "/opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks",
    "elastic.https.ssl.truststore.password": "yOb0eqkAqtj8HEWebgA7nf04YlqsLw44",
    "elastic.https.ssl.truststore.type": "JKS",
    "elastic.https.ssl.keystore.location": "/opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12",
    "elastic.https.ssl.keystore.password": "8siDJx7zdDhhu5iMMZwnhZfTaGFSgCvh",
    "elastic.https.ssl.keystore.type": "PKCS12",
    "key.ignore": "true",
    "schema.ignore": "true",
    "behavior.on.null.values": "delete",
    "write.method": "upsert",
    "type.name": "_doc",
    "max.in.flight.requests": "1",
    "batch.size": "2000",
    "linger.ms": "100",
    "flush.timeout.ms": "60000",
    "flush.synchronously": "true",
    "behavior.on.malformed.documents": "warn",
    "drop.invalid.message": "true",
    "max.retries": "10",
    "retry.backoff.ms": "5000",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",
    "transforms": "routeTopic,addZone,addEnv",
    "transforms.routeTopic.type": "org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.routeTopic.regex": "identity_sau_main_dev_account_router",
    "transforms.routeTopic.replacement": "identity_universe_main_dev_account_router",
    "transforms.addZone.type": "org.apache.kafka.connect.transforms.InsertField$Value",
    "transforms.addZone.static.field": "zone",
    "transforms.addZone.static.value": "sau",
    "transforms.addEnv.type": "org.apache.kafka.connect.transforms.InsertField$Value",
    "transforms.addEnv.static.field": "env",
    "transforms.addEnv.static.value": "dev"
  },
  "tasks": [],
  "type": "sink"
}

🔍 Verifying connector status...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Universe Identity ES Sink Status
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Connector: RUNNING
  Task:      RUNNING
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Universe Identity ES Sink is running

🔍 Checking Universe ES index document count...
   Universe index identity_universe_main_dev_account_router: 0 documents
   Documents from zone 'sau': 0
ℹ️  No documents yet - may take a moment for initial sync
   Run 10-test-universe-identity-index.sh to verify end-to-end

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  DUAL-SINK PATTERN SETUP COMPLETE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Connector:    pg_identity_sau_to_universe_main_dev_es_sink
  Source Topic: identity_sau_main_dev_account_router
  Universe Index: identity_universe_main_dev_account_router
  Zone Field: zone: "sau"

  Query example (filter by zone):
  GET identity_universe_main_dev_account_router/_search
  { "query": { "term": { "zone": "sau" } } }
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

✅ Done.
[2026-01-02 11:59:46] ✅ Completed: 03-setup-es-universe-sink.sh
[2026-01-02 11:59:46] 
[2026-01-02 11:59:46] 🔧 Running: 04-test-cdc-pipelines.sh
[2026-01-02 11:59:46]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/04-test-cdc-pipelines.sh
[2026-01-02 11:59:46]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
ℹ️  Using test identifier with hostname: coordinator-web-03
ℹ️  Citus: Data will be stored on coordinator
ℹ️  Test data identifier: coordinator
   - Username: cdc_user_coordinator_1767355193
   - This reflects the actual Citus storage node in Elasticsearch
→ Inserting into Postgres (auth.login_account)
   INSERT INTO auth.login_account (id, email, username, password_hash, status, region_hint, created_at, updated_at)
   VALUES ('019b7e94-199a-706e-8840-f49f0145b095', 'cdc_test_coordinator_1767355193@example.com', 'cdc_user_coordinator_1767355193', crypt('testpass123', gen_salt('bf')), 'active', 'sau', NOW(), NOW())
   RETURNING id, email, username, created_at;
                  id                  |                    email                    |            username             |          created_at           
--------------------------------------+---------------------------------------------+---------------------------------+-------------------------------
 019b7e94-199a-706e-8840-f49f0145b095 | cdc_test_coordinator_1767355193@example.com | cdc_user_coordinator_1767355193 | 2026-01-02 11:59:54.049271+00
(1 row)

INSERT 0 1
✅ Inserted test record: cdc_test_coordinator_1767355193@example.com
ℹ️  Citus placement: Shard 102024 on db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432
→ Waiting for CDC to propagate to Elasticsearch (index: identity_sau_main_dev_account_router, max 45s)

   Polling... elapsed: 0s/45s
   Polling... elapsed: 3s/45s
   ✅ Document found after 3s!                    
→ Final search: https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/identity_sau_main_dev_account_router/_search?q=email:cdc_test_coordinator_1767355193@example.com
🎉 SUCCESS: Document is indexed in 'identity_sau_main_dev_account_router'
📄 Indexed Document (source):
{
  "mysql_id": null,
  "__ts_ms": 1767355194235,
  "updated_at": "2026-01-02T11:59:54.049271Z",
  "password_hash": "$2a$06$ZQ.5D1WeJ4Iuo7.FkzHbCOIWRqS5KcTPuE6MGnw94qbBHnCRImieG",
  "created_at": "2026-01-02T11:59:54.049271Z",
  "id": "019b7e94-199a-706e-8840-f49f0145b095",
  "region_hint": "sau",
  "__op": "c",
  "email": "cdc_test_coordinator_1767355193@example.com",
  "username": "cdc_user_coordinator_1767355193",
  "status": "active"
}

📊 Elasticsearch Indices:
   health status index                                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
   green  open   identity_sau_main_dev_account_router-000001 TYpyHjYmSLeo-_L8xAAjJA   1   0          1            0      9.3kb          9.3kb        9.3kb
   
   [es-http] code=200 url=https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cat/indices?v time=0.079967s
[2026-01-02 11:59:57] ✅ Completed: 04-test-cdc-pipelines.sh
[2026-01-02 11:59:57] 
[2026-01-02 11:59:57] 🔧 Running: 05-verify-cdc-publication.sh
[2026-01-02 11:59:57]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/05-verify-cdc-publication.sh
[2026-01-02 11:59:57]    Executing directly (script is executable)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Verifying CDC Publication & Replication Slot
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔍 Checking CDC configuration...

1️⃣  Checking publication: cdc_pub_authn
   ⚠️  Publication does not exist - creating...
CREATE PUBLICATION
   ✅ Publication created successfully (with publish_via_partition_root = true)
 schemaname |   tablename   
------------+---------------
 auth       | login_account
(1 row)


2️⃣  Checking replication slot: slot_authn_sau_main_dev
   ⚠️  Replication slot does not exist
   ℹ️  This will be created automatically by Debezium when it connects

3️⃣  Checking table replica identity for CDC
   ℹ️  Replica identity: DEFAULT (only primary key changes)

   💡 CDC requires replica identity FULL for complete change capture
   🔧 Setting replica identity to FULL automatically...
ALTER TABLE
   ✅ Replica identity set to FULL

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ CDC Publication Verification Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📊 Summary:
  • Publication: ✅ Created
  • Replication Slot: ⚠️  Will be created by Debezium
  • Replica Identity: f

🔗 Next Steps:
  1. Check Kafka Connect status: curl -s https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_identity_sau_main_dev_debezium_postgres/status | jq
  2. Monitor Kafka topic: Check /var/lib/kafka/authN-af-aaaa1-dev*/authN_af_aaaa1_dev_account_router*/
  3. Verify Elasticsearch: Check the dashboard for real-time updates

[2026-01-02 12:00:03] ✅ Completed: 05-verify-cdc-publication.sh
[2026-01-02 12:00:03] 
[2026-01-02 12:00:04] 🔧 Running: 06-verify-cdc.sh
[2026-01-02 12:00:04]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/06-verify-cdc.sh
[2026-01-02 12:00:04]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  CDC Pipeline Verification
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔧 STEP 1: Testing CDC pipeline...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
ℹ️  Using test identifier with hostname: coordinator-web-03
parse error: Invalid numeric literal at line 2, column 9
parse error: Invalid numeric literal at line 2, column 9
⚠️  Could not resolve concrete index; will search base 'identity_sau_main_dev_account_router' (may fail if missing).
ℹ️  Citus: Data will be stored on coordinator
ℹ️  Test data identifier: coordinator
   - Username: cdc_user_coordinator_1767355494
   - This reflects the actual Citus storage node in Elasticsearch
→ Inserting into Postgres (auth.login_account)
   INSERT INTO auth.login_account (id, email, username, password_hash, status, region_hint, created_at, updated_at)
   VALUES ('019b7e98-b000-7041-a7c0-74c01c0f7c56', 'cdc_test_coordinator_1767355494@example.com', 'cdc_user_coordinator_1767355494', crypt('testpass123', gen_salt('bf')), 'active', 'sau', NOW(), NOW())
   RETURNING id, email, username, created_at;
                  id                  |                    email                    |            username             |          created_at           
--------------------------------------+---------------------------------------------+---------------------------------+-------------------------------
 019b7e98-b000-7041-a7c0-74c01c0f7c56 | cdc_test_coordinator_1767355494@example.com | cdc_user_coordinator_1767355494 | 2026-01-02 12:04:54.997379+00
(1 row)

INSERT 0 1
✅ Inserted test record: cdc_test_coordinator_1767355494@example.com
ℹ️  Citus placement: Shard 102024 on db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432
→ Waiting for CDC to propagate to Elasticsearch (index: identity_sau_main_dev_account_router, max 45s)

   Polling... elapsed: 0s/45s
   Polling... elapsed: 3s/45s
   Polling... elapsed: 6s/45s
   Polling... elapsed: 9s/45s
   Polling... elapsed: 12s/45s
   Polling... elapsed: 15s/45s
   Polling... elapsed: 18s/45s
   Polling... elapsed: 21s/45s
   Polling... elapsed: 24s/45s
   Polling... elapsed: 27s/45s
   Polling... elapsed: 30s/45s
   Polling... elapsed: 33s/45s
   Polling... elapsed: 36s/45s
   Polling... elapsed: 39s/45s
   Polling... elapsed: 42s/45s→ Final search: https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/identity_sau_main_dev_account_router/_search?q=email:cdc_test_coordinator_1767355494@example.com

   ⏱️  Timeout after 45s - document not found        
❌ FAIL: Document not found in 'identity_sau_main_dev_account_router'
ES search response (first lines):
{
  "took": 183,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}

→ Checking if index exists:
health status index                                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
green  open   identity_sau_main_dev_account_router-000001 TYpyHjYmSLeo-_L8xAAjJA   1   0          1            0      9.3kb          9.3kb        9.3kb

→ Document count in index:
{
  "count": 1,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  }
}

→ All Indices:
   health status index                                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
   green  open   identity_sau_main_dev_account_router-000001 TYpyHjYmSLeo-_L8xAAjJA   1   0          1            0      9.3kb          9.3kb        9.3kb
   
   [es-http] code=200 url=https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cat/indices?v time=0.075692s
❌ Error at line 36: bash "${SCRIPT_DIR}/04-test-cdc-pipelines.sh" "$IDENTIFIER" "$IDENTIFIER_PARENT" (exit 1)
[2026-01-02 12:05:42] ❌ FAILED: 06-verify-cdc.sh (exit code: 1)
[2026-01-02 12:05:42] ⚠️  Continuing with next step despite failure (non-critical step)...
[2026-01-02 12:05:42] 
[2026-01-02 12:05:42] 🔧 Running: 07-test-universe-identity-index.sh
[2026-01-02 12:05:42]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/07-test-universe-identity-index.sh
[2026-01-02 12:05:42]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Universe Identity Index Health Test
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment:     identity-sau-main-dev
  PostgreSQL:      db-identity-sau-main-dev-postgresql.fastorder.com:5432
  Zonal ES:     search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
  Universe ES:       search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
  Zonal Index:  identity_sau_main_dev_account_router
  Universe Index:    identity_universe_main_dev_account_router
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔐 Retrieving credentials...
❌ Could not retrieve PostgreSQL password
[2026-01-02 12:06:07] ❌ FAILED: 07-test-universe-identity-index.sh (exit code: 1)
[2026-01-02 12:06:07] ⚠️  Continuing with next step despite failure (non-critical step)...
[2026-01-02 12:06:07] 
[2026-01-02 12:06:07] ==========================================
[2026-01-02 12:06:07] ✅ CDC Pipeline setup complete for 1 subservice(s)
[2026-01-02 12:06:07] CDC SETUP SCRIPT FINISHED
[2026-01-02 12:06:07] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260102_115453.log
[2026-01-02 12:06:07] ==========================================
✓ ✅ CDC Pipeline setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 11-monitoring-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up monitoring for coordinator...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 PostgreSQL Monitoring Integration for identity-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[OK]   ✓ Observability cell is ready

[INFO] ✓ Using private IP for metrics: 10.100.1.213
[INFO] 2️⃣ Setting up postgres_exporter integration...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[INFO] Setting up postgres_exporter for identity-sau-main-dev
[2026-01-02 12:06:10 UTC] USER=www-data EUID=0 PID=2077741 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-identity-sau-main-dev.yaml /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[2026-01-02 12:06:10 UTC] USER=www-data EUID=0 PID=2077751 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[2026-01-02 12:06:10 UTC] USER=www-data EUID=0 PID=2077761 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[OK]   Custom queries file created at /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[2026-01-02 12:06:10 UTC] USER=www-data EUID=0 PID=2077771 ACTION=passthru ARGS=mv /tmp/postgres_exporter-identity-sau-main-dev.service /etc/systemd/system/postgres_exporter-identity-sau-main-dev.service
[2026-01-02 12:06:10 UTC] USER=www-data EUID=0 PID=2077780 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 12:06:11 UTC] USER=www-data EUID=0 PID=2077829 ACTION=passthru ARGS=systemctl enable postgres_exporter-identity-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/postgres_exporter-identity-sau-main-dev.service -> /etc/systemd/system/postgres_exporter-identity-sau-main-dev.service.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  IP Conflict Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: identity-sau-main-dev
IP Address:  10.100.1.213
Port:        9187
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔍 Checking IP conflict for identity-sau-main-dev on 10.100.1.213:9187...
✅ IP 10.100.1.213:9187 is available - no conflicts detected

🔍 Checking for orphaned processes that might conflict...
✅ No orphaned processes detected

✅ All checks passed - safe to proceed with identity-sau-main-dev setup
[2026-01-02 12:06:12 UTC] USER=www-data EUID=0 PID=2077924 ACTION=passthru ARGS=systemctl restart postgres_exporter-identity-sau-main-dev.service
[OK]   postgres_exporter configured on db-identity-sau-main-dev-postgresql.fastorder.com:9187
[INFO] Adding PostgreSQL scrape target to Prometheus config...
[OK]   PostgreSQL scrape target added
[INFO] Creating PostgreSQL alert rules...
[2026-01-02 12:06:14 UTC] USER=www-data EUID=0 PID=2077962 ACTION=fsop ARGS=mv /tmp/postgresql_alerts_identity-sau-main-dev.yml /etc/prometheus/obs-identity-sau-main-dev/rules/postgresql_alerts.yml
[OK]   PostgreSQL alert rules created: /etc/prometheus/obs-identity-sau-main-dev/rules/postgresql_alerts.yml
[INFO] Adding PostgreSQL alerts to Prometheus config...
[2026-01-02 12:06:15 UTC] USER=www-data EUID=0 PID=2077972 ACTION=fsop ARGS=sed -i /rule_files:/a\  - "rules/postgresql_alerts.yml" /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[OK]   PostgreSQL alerts registered in Prometheus
[2026-01-02 12:06:15 UTC] USER=www-data EUID=0 PID=2077982 ACTION=passthru ARGS=systemctl reload prometheus-obs-identity-sau-main-dev.service
Failed to reload prometheus-obs-identity-sau-main-dev.service: Job type reload is not applicable for unit prometheus-obs-identity-sau-main-dev.service.
[2026-01-02 12:06:15 UTC] USER=www-data EUID=0 PID=2077991 ACTION=passthru ARGS=systemctl restart prometheus-obs-identity-sau-main-dev.service
[OK]   Prometheus reloaded with PostgreSQL monitoring
[OK]   ✓ postgres_exporter integration complete
[INFO] Registering postgres_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: postgres_exporter -> 10.100.1.213:9187
[OK]   ✓ Registered postgres_exporter scrape target: 10.100.1.213:9187
[INFO]   Target file: /etc/prometheus/obs-identity-sau-main-dev/targets/postgres_exporter.yml
[OK]   ✓ postgres_exporter registered as Prometheus scrape target

[INFO] 3️⃣ Setting up pgbouncer_exporter integration...
[INFO] PgBouncer FQDN found in /etc/hosts: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.204
[INFO] PgBouncer detected: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com:6432
[OK]   ✓ pgbouncer_exporter already installed
[INFO] Getting pgbouncer_admin password (SERVICE=identity, ZONE=sau)
[OK]   ✓ pgbouncer_admin password retrieved (24 chars)
[INFO] Using pgbouncer certs from: /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[INFO] Creating pgbouncer_exporter systemd service...
[OK]   ✓ pgbouncer_exporter service file created
[INFO] Starting pgbouncer_exporter service...
[2026-01-02 12:06:17 UTC] USER=www-data EUID=0 PID=2078055 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 12:06:18 UTC] USER=www-data EUID=0 PID=2078100 ACTION=passthru ARGS=systemctl enable pgbouncer_exporter-identity-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/pgbouncer_exporter-identity-sau-main-dev.service -> /etc/systemd/system/pgbouncer_exporter-identity-sau-main-dev.service.
[2026-01-02 12:06:18 UTC] USER=www-data EUID=0 PID=2078150 ACTION=passthru ARGS=systemctl restart pgbouncer_exporter-identity-sau-main-dev.service
[WARN] ⚠️  pgbouncer_exporter service not running (may need manual start)
[WARN]     Run: systemctl status pgbouncer_exporter-identity-sau-main-dev.service

[INFO] 4️⃣ Registering nodes to monitoring database...
[INFO] PostgreSQL key permissions set for www-data access: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[INFO] Registering PostgreSQL coordinator to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.213
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: ce097707-5ce5-40c8-a941-01512555cab8
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ PostgreSQL coordinator registered
[INFO] Registering PgBouncer to monitoring dashboard...
[INFO]   FQDN: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com, IP: 10.100.1.204, Port: 6432
[INFO]   Key permissions set for www-data access
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        identity-sau-main-dev-pgbouncer
[INFO]   Identifier Parent: pooling
[INFO]   IP:                10.100.1.204
[INFO]   Port:              6432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 426480a5-2f64-4fc0-b2b5-710f9ccb059a
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ PgBouncer registered

[INFO] 5️⃣ Creating PgBouncer professional monitoring rules...
[INFO] Creating PgBouncer recording rules...
[OK]   ✓ PgBouncer recording rules created
[INFO] Creating PgBouncer alert rules with runbook URLs...
[OK]   ✓ PgBouncer alert rules with runbook URLs created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ PostgreSQL & PgBouncer Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] postgres_exporter: http://localhost:9187/metrics
[INFO] pgbouncer_exporter: http://localhost:9127/metrics
[INFO] Prometheus: https://metrics-identity-sau-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-identity-sau-main-dev.fastorder.com
[INFO] 
[INFO] PgBouncer Monitoring:
[INFO]   • Recording rules: /etc/prometheus/obs-identity-sau-main-dev/rules/pgbouncer_recording_rules.yml
[INFO]   • Alert rules: /etc/prometheus/obs-identity-sau-main-dev/rules/pgbouncer_alerts.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up monitoring for 1 worker(s) and 1 standby(s) per worker...
[INFO] Setting up monitoring for: worker-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 PostgreSQL Monitoring Integration for identity-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[OK]   ✓ Observability cell is ready

[INFO] ✓ Using private IP for metrics: 10.100.1.213
[INFO] 2️⃣ Setting up postgres_exporter integration...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[INFO] Setting up postgres_exporter for identity-sau-main-dev
[2026-01-02 12:06:24 UTC] USER=www-data EUID=0 PID=2078335 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-identity-sau-main-dev.yaml /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[2026-01-02 12:06:24 UTC] USER=www-data EUID=0 PID=2078344 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[2026-01-02 12:06:24 UTC] USER=www-data EUID=0 PID=2078353 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[OK]   Custom queries file created at /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[OK]   postgres_exporter already running with custom queries for identity-sau-main-dev
[OK]   ✓ postgres_exporter integration complete
[INFO] Registering postgres_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: postgres_exporter -> 10.100.1.213:9187
[OK]   ✓ Registered postgres_exporter scrape target: 10.100.1.213:9187
[INFO]   Target file: /etc/prometheus/obs-identity-sau-main-dev/targets/postgres_exporter.yml
[OK]   ✓ postgres_exporter registered as Prometheus scrape target

[INFO] 3️⃣ Setting up pgbouncer_exporter integration...
[INFO] PgBouncer FQDN found in /etc/hosts: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.204
[INFO] PgBouncer detected: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com:6432
[OK]   ✓ pgbouncer_exporter already installed
[INFO] Getting pgbouncer_admin password (SERVICE=identity, ZONE=sau)
[OK]   ✓ pgbouncer_admin password retrieved (24 chars)
[INFO] Using pgbouncer certs from: /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[INFO] Creating pgbouncer_exporter systemd service...
[OK]   ✓ pgbouncer_exporter service file created
[INFO] Starting pgbouncer_exporter service...
[2026-01-02 12:06:26 UTC] USER=www-data EUID=0 PID=2078411 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 12:06:27 UTC] USER=www-data EUID=0 PID=2078461 ACTION=passthru ARGS=systemctl enable pgbouncer_exporter-identity-sau-main-dev.service
[2026-01-02 12:06:28 UTC] USER=www-data EUID=0 PID=2078507 ACTION=passthru ARGS=systemctl restart pgbouncer_exporter-identity-sau-main-dev.service
[WARN] ⚠️  pgbouncer_exporter service not running (may need manual start)
[WARN]     Run: systemctl status pgbouncer_exporter-identity-sau-main-dev.service

[INFO] 4️⃣ Registering nodes to monitoring database...
[INFO] PostgreSQL key permissions set for www-data access: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[INFO] Registering PostgreSQL worker-01 to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-worker-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.213
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 2a8d7237-0c1b-4286-8ffc-cd46f4f7052e
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ PostgreSQL worker-01 registered
[INFO] Registering PgBouncer to monitoring dashboard...
[INFO]   FQDN: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com, IP: 10.100.1.204, Port: 6432
[INFO]   Key permissions set for www-data access
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        identity-sau-main-dev-pgbouncer
[INFO]   Identifier Parent: pooling
[INFO]   IP:                10.100.1.204
[INFO]   Port:              6432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 426480a5-2f64-4fc0-b2b5-710f9ccb059a
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ PgBouncer registered

[INFO] 5️⃣ Creating PgBouncer professional monitoring rules...
[INFO] Creating PgBouncer recording rules...
[OK]   ✓ PgBouncer recording rules created
[INFO] Creating PgBouncer alert rules with runbook URLs...
[OK]   ✓ PgBouncer alert rules with runbook URLs created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ PostgreSQL & PgBouncer Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] postgres_exporter: http://localhost:9187/metrics
[INFO] pgbouncer_exporter: http://localhost:9127/metrics
[INFO] Prometheus: https://metrics-identity-sau-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-identity-sau-main-dev.fastorder.com
[INFO] 
[INFO] PgBouncer Monitoring:
[INFO]   • Recording rules: /etc/prometheus/obs-identity-sau-main-dev/rules/pgbouncer_recording_rules.yml
[INFO]   • Alert rules: /etc/prometheus/obs-identity-sau-main-dev/rules/pgbouncer_alerts.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up monitoring for standby: worker-01-standby-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 PostgreSQL Monitoring Integration for identity-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[OK]   ✓ Observability cell is ready

[INFO] ✓ Using private IP for metrics: 10.100.1.213
[INFO] 2️⃣ Setting up postgres_exporter integration...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[INFO] Setting up postgres_exporter for identity-sau-main-dev
[2026-01-02 12:06:33 UTC] USER=www-data EUID=0 PID=2078705 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-identity-sau-main-dev.yaml /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[2026-01-02 12:06:33 UTC] USER=www-data EUID=0 PID=2078714 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[2026-01-02 12:06:33 UTC] USER=www-data EUID=0 PID=2078723 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[OK]   Custom queries file created at /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[OK]   postgres_exporter already running with custom queries for identity-sau-main-dev
[OK]   ✓ postgres_exporter integration complete
[INFO] Registering postgres_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: postgres_exporter -> 10.100.1.213:9187
[OK]   ✓ Registered postgres_exporter scrape target: 10.100.1.213:9187
[INFO]   Target file: /etc/prometheus/obs-identity-sau-main-dev/targets/postgres_exporter.yml
[OK]   ✓ postgres_exporter registered as Prometheus scrape target

[INFO] 3️⃣ Setting up pgbouncer_exporter integration...
[INFO] PgBouncer FQDN found in /etc/hosts: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.204
[INFO] PgBouncer detected: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com:6432
[OK]   ✓ pgbouncer_exporter already installed
[INFO] Getting pgbouncer_admin password (SERVICE=identity, ZONE=sau)
[OK]   ✓ pgbouncer_admin password retrieved (24 chars)
[INFO] Using pgbouncer certs from: /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[INFO] Creating pgbouncer_exporter systemd service...
[OK]   ✓ pgbouncer_exporter service file created
[INFO] Starting pgbouncer_exporter service...
[2026-01-02 12:06:36 UTC] USER=www-data EUID=0 PID=2078780 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 12:06:37 UTC] USER=www-data EUID=0 PID=2078833 ACTION=passthru ARGS=systemctl enable pgbouncer_exporter-identity-sau-main-dev.service
[2026-01-02 12:06:38 UTC] USER=www-data EUID=0 PID=2078878 ACTION=passthru ARGS=systemctl restart pgbouncer_exporter-identity-sau-main-dev.service
[WARN] ⚠️  pgbouncer_exporter service not running (may need manual start)
[WARN]     Run: systemctl status pgbouncer_exporter-identity-sau-main-dev.service

[INFO] 4️⃣ Registering nodes to monitoring database...
[INFO] Registering PostgreSQL worker-01-standby-01 to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-worker-01-standby-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.213
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 8eaa8059-bede-4f71-ae1d-d26590a898da
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ PostgreSQL worker-01-standby-01 registered
[INFO] Registering PgBouncer to monitoring dashboard...
[INFO]   FQDN: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com, IP: 10.100.1.204, Port: 6432
[INFO]   Key permissions set for www-data access
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        identity-sau-main-dev-pgbouncer
[INFO]   Identifier Parent: pooling
[INFO]   IP:                10.100.1.204
[INFO]   Port:              6432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 426480a5-2f64-4fc0-b2b5-710f9ccb059a
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ PgBouncer registered

[INFO] 5️⃣ Creating PgBouncer professional monitoring rules...
[INFO] Creating PgBouncer recording rules...
[OK]   ✓ PgBouncer recording rules created
[INFO] Creating PgBouncer alert rules with runbook URLs...
[OK]   ✓ PgBouncer alert rules with runbook URLs created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ PostgreSQL & PgBouncer Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] postgres_exporter: http://localhost:9187/metrics
[INFO] pgbouncer_exporter: http://localhost:9127/metrics
[INFO] Prometheus: https://metrics-identity-sau-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-identity-sau-main-dev.fastorder.com
[INFO] 
[INFO] PgBouncer Monitoring:
[INFO]   • Recording rules: /etc/prometheus/obs-identity-sau-main-dev/rules/pgbouncer_recording_rules.yml
[INFO]   • Alert rules: /etc/prometheus/obs-identity-sau-main-dev/rules/pgbouncer_alerts.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ Monitoring setup completed for coordinator, workers, and standbys

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 12-setup-offsite-backup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Setting up offsite backup repository for identity-sau-main-dev...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Offsite Backup Repository Setup (repo2)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📋 OFFSITE BACKUP INFORMATION
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Why Offsite Backups?
[INFO]   ✓ Disaster recovery resilience (datacenter loss, hardware failure)
[INFO]   ✓ Protection against local corruption or ransomware
[INFO]   ✓ Compliance requirements (geographic redundancy)
[INFO]   ✓ Long-term archival with cost-effective storage tiers

[WARN] ⚠️  Offsite backup (repo2) is NOT ENABLED
[WARN]    Using local backups only (repo1)

[INFO] Configuration Example Location:
[INFO]   📄 /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example

[INFO] Supported Storage Backends:
[INFO]   • AWS S3 (standard, multi-region)
[INFO]   • AWS S3 Glacier (low-cost archival)
[INFO]   • MinIO (self-hosted S3-compatible)
[INFO]   • Google Cloud Storage (via S3 compatibility)
[INFO]   • Azure Blob Storage (via S3 compatibility)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📝 SETUP INSTRUCTIONS
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Step 1: Review the example configuration
[INFO]   cat /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example

[INFO] Step 2: Prepare S3 bucket and credentials
[INFO]   • Create S3 bucket (or MinIO bucket)
[INFO]   • Create IAM user with S3 permissions (PutObject, GetObject, DeleteObject, ListBucket)
[INFO]   • Note: Access Key ID and Secret Access Key

[INFO] Step 3: Add repo2 configuration to /etc/pgbackrest/pgbackrest.conf
[INFO]   • Copy repo2-* settings from example to [global] section
[INFO]   • Replace placeholders (bucket name, access keys, region)
[INFO]   • Note: Use same cipher key as repo1, or generate separate key for repo2

[INFO] Step 4: Initialize repo2 stanzas
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator stanza-create --repo=2
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-worker-01 stanza-create --repo=2
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-worker-02 stanza-create --repo=2

[INFO] Step 5: Verify repo2 configuration
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator check --repo=2

[INFO] Step 6: Take initial full backup to repo2
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator --repo=2 --type=full backup

[INFO] Step 7: Update backup automation to include repo2
[INFO]   • Edit: /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[INFO]   • Change: pgbackrest backup to pgbackrest --repo=1,2 backup
[INFO]   • Or: Add separate cron for repo2 backups

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🧪 TESTING
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] After configuration, run:
[INFO]   ./08-setup-offsite-backup.sh test

[INFO] This will verify:
[INFO]   ✓ S3 connectivity
[INFO]   ✓ Stanza initialization
[INFO]   ✓ Test backup and restore from repo2

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 💡 COST OPTIMIZATION
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] AWS S3 Lifecycle Policies (transition to cheaper storage):
[INFO]   • 0-30 days:   S3 Standard (~$0.023/GB/month)
[INFO]   • 30-90 days:  S3 Standard-IA (~$0.0125/GB/month)
[INFO]   • 90+ days:    S3 Glacier (~$0.004/GB/month)

[INFO] Estimated costs for 100GB backups:
[INFO]   • All Standard:     ~$2.30/month
[INFO]   • With lifecycle:   ~$1.20/month (48% savings)


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 13-setup-monitoring-alerts.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Setting up backup monitoring and alerting for identity-sau-main-dev...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Monitoring and Alerting Configuration
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] This will set up monitoring for:
  • Backup failures (cron job failures)
  • WAL archiving backlog (>100 files)
  • Repository disk space (<20% free)
  • Backup age (>25 hours)

[INFO] No alert email configured (set ALERT_EMAIL environment variable)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Creating monitoring directories...
[2026-01-02 12:06:47 UTC] USER=www-data EUID=0 PID=2079428 ACTION=fsop ARGS=mkdir -p /opt/pgbackrest-monitoring
[2026-01-02 12:06:47 UTC] USER=www-data EUID=0 PID=2079437 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest-monitoring
[2026-01-02 12:06:47 UTC] USER=www-data EUID=0 PID=2079446 ACTION=fsop ARGS=chmod 777 /opt/pgbackrest-monitoring
[2026-01-02 12:06:47 UTC] USER=www-data EUID=0 PID=2079455 ACTION=fsop ARGS=chmod 777 /var/log/pgbackrest-monitoring
[2026-01-02 12:06:47 UTC] USER=www-data EUID=0 PID=2079464 ACTION=fsop ARGS=chown postgres:postgres /opt/pgbackrest-monitoring
[2026-01-02 12:06:47 UTC] USER=www-data EUID=0 PID=2079473 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest-monitoring
[INFO] ✅ Directories created

[INFO] 2️⃣ Creating alert helper script...
[2026-01-02 12:06:47 UTC] USER=www-data EUID=0 PID=2079493 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/send-alert.sh
[INFO] ✅ Alert helper created

[INFO] 3️⃣ Creating WAL queue monitoring script...
[2026-01-02 12:06:47 UTC] USER=www-data EUID=0 PID=2079513 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-wal-queue.sh
[INFO] ✅ WAL queue monitor created

[INFO] 4️⃣ Creating backup age monitoring script...
[2026-01-02 12:06:47 UTC] USER=www-data EUID=0 PID=2079533 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-backup-age.sh
[INFO] ✅ Backup age monitor created

[INFO] 5️⃣ Creating repository disk space monitoring script...
[2026-01-02 12:06:47 UTC] USER=www-data EUID=0 PID=2079552 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-repo-space.sh
[INFO] ✅ Disk space monitor created

[INFO] 6️⃣ Creating backup failure detection script...
[2026-01-02 12:06:47 UTC] USER=www-data EUID=0 PID=2079572 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-backup-failures.sh
[INFO] ✅ Backup failure detector created

[INFO] 7️⃣ Creating master monitoring script...
[2026-01-02 12:06:47 UTC] USER=www-data EUID=0 PID=2079590 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO] ✅ Master monitoring script created

[INFO] 8️⃣ Installing mailutils for email alerts...
[INFO] ✅ mailutils already installed

[INFO] 9️⃣ Installing jq for JSON parsing...
[INFO] ✅ jq already installed

[INFO] 🔟 Setting up monitoring cron jobs...
[2026-01-02 12:06:48 UTC] USER=www-data EUID=0 PID=2079608 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-monitoring-identity-sau-main-dev
[INFO] ✅ Monitoring cron jobs configured
[INFO]    Checks run every 15 minutes

[INFO] 1️⃣1️⃣ Creating monitoring dashboard...
[2026-01-02 12:06:48 UTC] USER=www-data EUID=0 PID=2079628 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/dashboard.sh
[INFO] ✅ Monitoring dashboard created

[INFO] 1️⃣2️⃣ Running initial monitoring check...

[2026-01-02 12:06:48 UTC] USER=www-data EUID=0 PID=2079637 ACTION=passthru ARGS=bash /opt/pgbackrest-monitoring/run-all-checks.sh
grep: write error: Broken pipe

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup monitoring setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Monitoring Configuration:
[INFO]   Alert Email:        
[INFO]   Slack Webhook:      Not configured

[INFO] Monitoring Checks:
[INFO]   • WAL Queue:        Every 15 minutes (threshold: >100 files)
[INFO]   • Backup Age:       Every 15 minutes (threshold: >25 hours)
[INFO]   • Disk Space:       Every 15 minutes (threshold: <20% free)
[INFO]   • Backup Failures:  Every 15 minutes (log analysis)

[INFO] Scripts Created:
[INFO]   Monitoring dir:     /opt/pgbackrest-monitoring
[INFO]   Log dir:            /var/log/pgbackrest-monitoring
[INFO]   Dashboard:          /opt/pgbackrest-monitoring/dashboard.sh
[INFO]   Master check:       /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO]   Alert sender:       /opt/pgbackrest-monitoring/send-alert.sh

[INFO] Useful Commands:
[INFO]   View dashboard:     /usr/local/bin/fastorder-provisioning-wrapper.sh /opt/pgbackrest-monitoring/dashboard.sh
[INFO]   Run checks now:     /usr/local/bin/fastorder-provisioning-wrapper.sh /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO]   View alerts:        tail -f /var/log/pgbackrest-monitoring/alerts.log
[INFO]   View monitoring:    tail -f /var/log/pgbackrest-monitoring/monitoring.log

[INFO] Cron Schedule:
[INFO]   All checks:         Every 15 minutes
[INFO]   Log rotation:       Weekly (keep 7 days)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 14-vault-cipher-key.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ✅ Using permanent AWS credentials from /home/ab/.aws/credentials [default] profile
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔐 PostgreSQL Cipher Key Vaulting
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Environment:       identity-sau-main-dev
[INFO]   AWS Region:        me-central-1
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣  Verifying AWS setup...
[INFO] ✅ AWS authentication successful

[INFO] 2️⃣  Verifying cipher key...
[INFO] ✅ Cipher key found
[INFO]    Location: /etc/pgbackrest/.cipher-key-identity-sau-main-dev
[INFO]    Hash (MD5): cb0ad657dd7d2644e3ad48e19928ee14
[INFO]    Size: 194 bytes

[INFO] 3️⃣  Vaulting cipher key to AWS Secrets Manager...
[INFO]    Secret name: fastorder/db/identity/sau/main/dev/postgresql/pgbackrest/cipher-key
[INFO]    Creating new secret...
[INFO] ✅ Cipher key stored in AWS Secrets Manager
[INFO]    Verifying storage...
[INFO] ✅ Verification successful - key matches

[INFO] 4️⃣  Creating local encrypted backup...
[2026-01-02 12:07:00 UTC] USER=www-data EUID=0 PID=2079895 ACTION=fsop ARGS=mv /tmp/cipher-key-backup-2079777.enc /root/.pgbackrest-cipher-key-identity-sau-main-dev.enc
[2026-01-02 12:07:00 UTC] USER=www-data EUID=0 PID=2079904 ACTION=fsop ARGS=chmod 600 /root/.pgbackrest-cipher-key-identity-sau-main-dev.enc
[2026-01-02 12:07:00 UTC] USER=www-data EUID=0 PID=2079925 ACTION=fsop ARGS=chmod 600 /root/.pgbackrest-cipher-key-passphrase-identity-sau-main-dev.txt
[INFO] ✅ Local encrypted backup created
[INFO]    Backup file: /root/.pgbackrest-cipher-key-identity-sau-main-dev.enc
[INFO]    Passphrase: /root/.pgbackrest-cipher-key-passphrase-identity-sau-main-dev.txt

[INFO] 5️⃣  Vaulting backup passphrase...
[INFO] ✅ Backup passphrase vaulted

[INFO] 6️⃣  Creating recovery documentation...
[2026-01-02 12:07:05 UTC] USER=www-data EUID=0 PID=2079978 ACTION=fsop ARGS=chmod 640 /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_identity-sau-main-dev.md
[2026-01-02 12:07:05 UTC] USER=www-data EUID=0 PID=2079987 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_identity-sau-main-dev.md
[INFO] ✅ Recovery documentation: /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_identity-sau-main-dev.md

[INFO] 7️⃣  Storing backup metadata...
[INFO] ✅ Backup metadata stored in AWS Secrets Manager
[INFO]    Secret: fastorder/db/identity/sau/main/dev/postgresql/backup/metadata-20260102-120705

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Cipher Key Vaulting Complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO]   Environment:          identity-sau-main-dev
[INFO]   Key Hash:             cb0ad657dd7d2644e3ad48e19928ee14

[INFO] AWS Secrets:
[INFO]   Cipher Key:           fastorder/db/identity/sau/main/dev/postgresql/pgbackrest/cipher-key
[INFO]   Passphrase:           fastorder/db/identity/sau/main/dev/postgresql/pgbackrest/cipher-key-passphrase
[INFO]   Backup Metadata:      fastorder/db/identity/sau/main/dev/postgresql/backup/metadata-20260102-120705

[INFO] Local Backups:
[INFO]   Encrypted File:       /root/.pgbackrest-cipher-key-identity-sau-main-dev.enc
[INFO]   Passphrase File:      /root/.pgbackrest-cipher-key-passphrase-identity-sau-main-dev.txt

[INFO] Recovery Doc:           /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_identity-sau-main-dev.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Skipping 15-backup-restore-test.sh (test script - set RUN_TESTS=true to enable)
[INFO] Skipping 16-test-recovery.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 17-verification.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)

[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] PostgreSQL Production Readiness Verification
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] 
[INFO] This script verifies 3 CRITICAL checks for production readiness:
[INFO]   1. Citus Cluster Operational (coordinator + workers)
[INFO]   2. SSL/TLS Enforced (certificates valid, connections secure)
[INFO]   3. Coordinator Backups Configured (pgBackRest functional)
[INFO] 
[INFO] 📖 Documentation: /tmp/VERIFICATION_RUNBOOK.md
[INFO] 🔐 Security: Uses sudo for certificate checks (maintains strict permissions)
[INFO] 📊 Exit Code: 0 = production ready, 1 = critical checks failed
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🕐 Ensuring all PostgreSQL services are ready...
[ OK ] ✅ All PostgreSQL services are ready

[INFO] 🔍 Starting PostgreSQL verification...
[INFO] Environment: identity-sau-main-dev
[INFO] Citus: yes

[INFO] Citus mode ENABLED
[INFO] → Coordinator + 1 worker(s) + 3 HA node(s) per worker

[INFO] Verifying 1 worker(s)...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying: worker-01 (type: worker-01)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Starting PostgreSQL verification for identity-sau-main-dev-worker-01...

[INFO] 1️⃣ Checking systemd service status...
[OK]   ✅ Service postgresql@identity-sau-main-dev-worker-01.service is active

[INFO] 2️⃣ Checking PostgreSQL process...
[OK]   ✅ PostgreSQL process is running

[INFO] 3️⃣ Checking socket directory...
[OK]   ✅ Socket directory exists: /var/run/postgresql-identity-sau-main-dev-worker-01
total 4
drwxrwsr-x  2 postgres postgres   80 Jan  2 11:41 .
drwxr-xr-x 56 root     root     1460 Jan  2 12:04 ..
srwxrwxrwx  1 postgres postgres    0 Jan  2 11:41 .s.PGSQL.5432
-rw-------  1 postgres postgres  131 Jan  2 11:41 .s.PGSQL.5432.lock

[INFO] 4️⃣ Testing connection via Unix socket...
[OK]   ✅ Socket connection successful
                                                              version                                                              
-----------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.6 (Ubuntu 17.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

[INFO] 5️⃣ Checking SSL certificates...
[2026-01-02 12:07:10 UTC] USER=www-data EUID=0 PID=2080128 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[OK]   ✅ Server certificate exists: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 12:07:10 UTC] USER=www-data EUID=0 PID=2080137 ACTION=fsop ARGS=openssl x509 -in /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt -noout -checkend 86400
Certificate will not expire
[OK]   ✅ Server certificate is valid
[2026-01-02 12:07:11 UTC] USER=www-data EUID=0 PID=2080146 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[OK]   ✅ CA certificate exists: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[INFO] ℹ️  Client certificates not found at /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[INFO]     (This is OK if using password authentication)

[INFO] 6️⃣ Checking PostgreSQL settings...
[OK]   ✅ SSL is enabled worker-01 worker-01
[OK]   ✅ Max connections: 100
[OK]   ✅ Listen addresses: 10.100.1.214
[OK]   ✅ WAL level: logical
[OK]   ✅ Shared preload libraries: shared_preload_libraries

[INFO] 7️⃣ Checking replication configuration...
[INFO] ℹ️  No synchronous standbys configured (single node or async replication)
[INFO] Checking replication slots...
      slot_name       | slot_type | active | restart_lsn 
----------------------+-----------+--------+-------------
 worker_01_standby_01 | physical  | f      | 
(1 row)
[OK]   ✅ Replication slot naming uses underscores (correct)
[INFO] Checking active replication connections...
 application_name | client_addr | state | sync_state 
------------------+-------------+-------+------------
(0 rows)
[INFO] ℹ️  No active replication connections
[INFO] ℹ️  This is a PRIMARY node (no standby.signal)

[INFO] 8️⃣ Checking pg_hba.conf for replication rules...
[WARN] ⚠️ pg_hba.conf not found at /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf

[INFO] 9️⃣ Checking Citus configuration...
[OK]   ✅ Citus extension is installed
[OK]   ✅ Citus version: Citus 13.2.0
[OK]   ✅ max_prepared_transactions: 100 (adequate for Citus)
[INFO] Citus active worker nodes:
                          node_name                          | node_port 
-------------------------------------------------------------+-----------
 db-identity-sau-main-dev-postgresql-worker-01.fastorder.com |      5432
(1 row)



[INFO] 🔟 Checking data directory...
[OK]   ✅ Data directory exists: /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[OK]   ✅ Data directory size: 4.0K

[INFO] 1️⃣1️⃣ Checking PgBouncer configuration...
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini' as root on web-03.
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/%i/pgbouncer.ini' as root on web-03.
[OK]   ✅ PgBouncer is installed
[INFO]    Version: 1.24.1
2.1.12-stable
c-ares
OpenSSL
yes
Failed to print table: Broken pipe
[INFO] ℹ️  PgBouncer service not configured for this environment

[INFO] 1️⃣2️⃣ Enhanced PgBouncer Admin Console Verification...
[INFO] ℹ️  PgBouncer password not found

[INFO] 1️⃣3️⃣ Replicator User Connection Verification...
[INFO] Found 1 replication slot(s) - verifying replicator connectivity...
[WARN] ⚠️ Replicator certificates not found at /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[INFO]    Expected files:
[INFO]    - /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[INFO]    - /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[INFO]    - /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[INFO] Checking pg_hba.conf replicator rules...
[OK]   ✅ Replicator HBA rules found:
 line_number |  type   |   database    |  user_name   |   address    |  auth_method  | options | error 
-------------+---------+---------------+--------------+--------------+---------------+---------+-------
          20 | hostssl | {replication} | {replicator} | 10.100.1.211 | scram-sha-256 |         | 
          21 | hostssl | {replication} | {replicator} | 10.100.1.214 | scram-sha-256 |         | 
(2 rows)
[INFO] Checking active replicator connections in pg_stat_activity...
[WARN] ⚠️ No active replicator connections in pg_stat_activity
[WARN]    This is expected if standbys are not currently connected

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ PostgreSQL verification completed successfully!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Instance:       identity-sau-main-dev-worker-01
[INFO] Service:        postgresql@identity-sau-main-dev-worker-01.service
[INFO] Socket:         /var/run/postgresql-identity-sau-main-dev-worker-01
[INFO] Data Directory: /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[INFO] Hostname:       db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO] Port:           5432
[INFO] SSL:            on
[INFO] WAL Level:      logical
[INFO] Citus:          yes
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Failed to print table: Broken pipe

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 💡 OPTIMIZATION OPPORTUNITIES (Optional Enhancements)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1. Review connection limits for production workload
[INFO]    🔌 Current: max_connections = 100 (PostgreSQL default)
[INFO]    💡 Consider: Increasing to 200-500 for production applications
[INFO]    ⚙️  Alternative: Use PgBouncer connection pooling (lower PostgreSQL limit, higher client capacity)
[INFO]    🔧 Action: Adjust max_connections in postgresql.conf based on workload analysis
[INFO]    ⚠️  Note: Each connection consumes ~10MB RAM; tune based on available memory
[INFO]    📚 Docs: https://www.postgresql.org/docs/current/runtime-config-connection.html

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ℹ️  These are optional enhancements for production-scale deployments
[INFO] ℹ️  Current configuration is fully functional and ready for production
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] ✅ Verification passed for worker-01

Failed to print table: Broken pipe
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying: worker-01-standby-01 (type: worker-01)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Starting PostgreSQL verification for identity-sau-main-dev-worker-01-standby-01...

[INFO] 1️⃣ Checking systemd service status...
[OK]   ✅ Service postgresql@identity-sau-main-dev-worker-01-standby-01.service is active

[INFO] 2️⃣ Checking PostgreSQL process...
[OK]   ✅ PostgreSQL process is running

[INFO] 3️⃣ Checking socket directory...
[OK]   ✅ Socket directory exists: /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
total 4
drwxrwsr-x  2 postgres postgres   80 Jan  2 11:51 .
drwxr-xr-x 56 root     root     1460 Jan  2 12:04 ..
srwxrwxrwx  1 postgres postgres    0 Jan  2 11:51 .s.PGSQL.5432
-rw-------  1 postgres postgres  153 Jan  2 11:51 .s.PGSQL.5432.lock

[INFO] 4️⃣ Testing connection via Unix socket...
[OK]   ✅ Socket connection successful
                                                              version                                                              
-----------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.6 (Ubuntu 17.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

[INFO] 5️⃣ Checking SSL certificates...
[2026-01-02 12:07:34 UTC] USER=www-data EUID=0 PID=2080557 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[OK]   ✅ Server certificate exists: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 12:07:34 UTC] USER=www-data EUID=0 PID=2080566 ACTION=fsop ARGS=openssl x509 -in /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt -noout -checkend 86400
Certificate will not expire
[OK]   ✅ Server certificate is valid
[2026-01-02 12:07:34 UTC] USER=www-data EUID=0 PID=2080575 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[OK]   ✅ CA certificate exists: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[INFO] ℹ️  Client certificates not found at /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[INFO]     (This is OK if using password authentication)

[INFO] 6️⃣ Checking PostgreSQL settings...
[OK]   ✅ SSL is enabled worker-01 worker-01-standby-01
[OK]   ✅ Max connections: 100
[OK]   ✅ Listen addresses: 10.100.1.211
[OK]   ✅ WAL level: logical
[OK]   ✅ Shared preload libraries: shared_preload_libraries

[INFO] 7️⃣ Checking replication configuration...
[INFO] ℹ️  No synchronous standbys configured (single node or async replication)
[INFO] Checking replication slots...
 slot_name | slot_type | active | restart_lsn 
-----------+-----------+--------+-------------
(0 rows)
[OK]   ✅ Replication slot naming uses underscores (correct)
[INFO] Checking active replication connections...
 application_name | client_addr | state | sync_state 
------------------+-------------+-------+------------
(0 rows)
[INFO] ℹ️  No active replication connections
[INFO] ℹ️  This is a PRIMARY node (no standby.signal)

[INFO] 8️⃣ Checking pg_hba.conf for replication rules...
[WARN] ⚠️ pg_hba.conf not found at /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/pg_hba.conf

[INFO] 9️⃣ Checking Citus configuration...
[INFO] ℹ️  Citus extension not needed on standby (will inherit from primary via replication)

[INFO] 🔟 Checking data directory...
[OK]   ✅ Data directory exists: /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[OK]   ✅ Data directory size: 4.0K

[INFO] 1️⃣1️⃣ Checking PgBouncer configuration...
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini' as root on web-03.
Failed to print table: Broken pipe
[OK]   ✅ PgBouncer is installed
[INFO]    Version: 1.24.1
2.1.12-stable
c-ares
OpenSSL
yes
Failed to print table: Broken pipe
[INFO] ℹ️  PgBouncer service not configured for this environment

[INFO] 1️⃣2️⃣ Enhanced PgBouncer Admin Console Verification...
Failed to print table: Broken pipe
[INFO] ℹ️  PgBouncer not configured for enhanced verification

[INFO] 1️⃣3️⃣ Replicator User Connection Verification...
[INFO] ℹ️  No replication slots configured - skipping replicator verification

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ PostgreSQL verification completed successfully!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Instance:       identity-sau-main-dev-worker-01-standby-01
[INFO] Service:        postgresql@identity-sau-main-dev-worker-01-standby-01.service
[INFO] Socket:         /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[INFO] Data Directory: /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[INFO] Hostname:       db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
[INFO] Port:           5432
[INFO] SSL:            on
[INFO] WAL Level:      logical
[INFO] Citus:          yes
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Failed to print table: Broken pipe
[ OK ] ✅ Verification passed for worker-01-standby-01

[INFO] Skipping worker-01-standby-02 - service not configured
[INFO] Skipping worker-01-standby-03 - service not configured
[INFO] Verifying coordinator...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying: coordinator (type: coordinator)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Starting PostgreSQL verification for identity-sau-main-dev-coordinator...

[INFO] 1️⃣ Checking systemd service status...
[OK]   ✅ Service postgresql@identity-sau-main-dev-coordinator.service is active

[INFO] 2️⃣ Checking PostgreSQL process...
[OK]   ✅ PostgreSQL process is running

[INFO] 3️⃣ Checking socket directory...
[OK]   ✅ Socket directory exists: /var/run/postgresql-identity-sau-main-dev-coordinator
total 4
drwxrwsr-x  2 postgres postgres   80 Jan  2 11:54 .
drwxr-xr-x 56 root     root     1460 Jan  2 12:04 ..
srwxrwxrwx  1 postgres postgres    0 Jan  2 11:54 .s.PGSQL.5432
-rw-------  1 postgres postgres  135 Jan  2 11:54 .s.PGSQL.5432.lock

[INFO] 4️⃣ Testing connection via Unix socket...
[OK]   ✅ Socket connection successful
                                                              version                                                              
-----------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.6 (Ubuntu 17.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

[INFO] 5️⃣ Checking SSL certificates...
[2026-01-02 12:07:54 UTC] USER=www-data EUID=0 PID=2081281 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[OK]   ✅ Server certificate exists: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 12:07:54 UTC] USER=www-data EUID=0 PID=2081292 ACTION=fsop ARGS=openssl x509 -in /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt -noout -checkend 86400
Certificate will not expire
[OK]   ✅ Server certificate is valid
[2026-01-02 12:07:54 UTC] USER=www-data EUID=0 PID=2081301 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[OK]   ✅ CA certificate exists: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[INFO] ℹ️  Client certificates not found at /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[INFO]     (This is OK if using password authentication)

[INFO] 6️⃣ Checking PostgreSQL settings...
[OK]   ✅ SSL is enabled coordinator coordinator
[OK]   ✅ Max connections: 150
[OK]   ✅ Listen addresses: 10.100.1.213
[OK]   ✅ WAL level: logical
[OK]   ✅ Shared preload libraries: shared_preload_libraries

[INFO] 7️⃣ Checking replication configuration...
[INFO] ℹ️  No synchronous standbys configured (single node or async replication)
[INFO] Checking replication slots...
         slot_name          | slot_type | active | restart_lsn 
----------------------------+-----------+--------+-------------
 slot_identity_sau_main_dev | logical   | t      | 0/8023260
(1 row)
[OK]   ✅ Replication slot naming uses underscores (correct)
[INFO] Checking active replication connections...
  application_name  | client_addr  |   state   | sync_state 
--------------------+--------------+-----------+------------
 Debezium Streaming | 10.100.1.213 | streaming | async
(1 row)
[INFO] ℹ️  Async replication is active
[INFO] ℹ️  This is a PRIMARY node (no standby.signal)

[INFO] 8️⃣ Checking pg_hba.conf for replication rules...
[WARN] ⚠️ pg_hba.conf not found at /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf

[INFO] 9️⃣ Checking Citus configuration...
[OK]   ✅ Citus extension is installed
[OK]   ✅ Citus version: Citus 13.2.0
[OK]   ✅ max_prepared_transactions: 100 (adequate for Citus)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] COORDINATOR-SPECIFIC CHECKS
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking registered workers...
[INFO] ℹ️  Coordinator role verified via pg_dist_node (1 workers registered)
[INFO] Checking coordinator hostname configuration...
[OK]   ✅ Coordinator hostname: ---------------------------------------------------------------:----------
[INFO] Checking for stuck prepared transactions...
[OK]   ✅ No stuck Citus prepared transactions
[INFO] Expected workers: 1
[INFO] Registered workers: 1
[OK]   ✅ All 1 worker(s) successfully registered
[INFO] Registered worker nodes:
                           nodename                            | nodeport | groupid | isactive | noderole | shouldhaveshards 
---------------------------------------------------------------+----------+---------+----------+----------+------------------
 db-identity-sau-main-dev-postgresql-coordinator.fastorder.com |     5432 |       0 | t        | primary  | f
 db-identity-sau-main-dev-postgresql-worker-01.fastorder.com   |     5432 |       1 | t        | primary  | t
(2 rows)

[INFO] Note: groupid=0 is the coordinator, groupid>0 are workers

[INFO] Citus active worker nodes:
                          node_name                          | node_port 
-------------------------------------------------------------+-----------
 db-identity-sau-main-dev-postgresql-worker-01.fastorder.com |      5432
(1 row)


[INFO] Verifying Citus workers...
[INFO] Checking worker: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
 citus_add_node 
----------------
              2
(1 row)


[INFO] Testing Citus distributed table setup...
[INFO] Checking for blocking locks...
    SELECT pg_terminate_backend(pid)
    FROM pg_stat_activity
    WHERE pid <> pg_backend_pid()
      AND state = 'idle in transaction'
      AND query_start < NOW() - INTERVAL '30 seconds'
      AND datname = current_database();
  
 pg_terminate_backend 
----------------------
(0 rows)

[INFO] Creating demo schema (if needed)...
CREATE SCHEMA
[OK]   ✅ Demo schema ready
[INFO] Creating distributed table 'demo.events'...
CREATE TABLE
[OK]   ✅ Table is already distributed
[INFO] Inserting test data...
INSERT 0 1
[OK]   ✅ Distributed table contains 1 row(s)
[INFO] Checking shard distribution...
[OK]   ✅ Table has 1 shard(s)
[INFO] Shard placement across workers (first 10 shards):
 shardid | nodename | nodeport | shardstate 
---------+----------+----------+------------
(0 rows)
[OK]   ✅ Verified 3 shard placement(s)
[INFO] Testing query routing (EXPLAIN for user_id=42)...
[INFO]    Query plan:         QUERY PLAN        
--------------------------
 Seq Scan on events
   Filter: (user_id = 42)
(2 rows)


[INFO] 🔟 Checking data directory...
[OK]   ✅ Data directory exists: /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[OK]   ✅ Data directory size: 4.0K

[INFO] 1️⃣1️⃣ Checking PgBouncer configuration...
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini' as root on web-03.
Failed to print table: Broken pipe
[OK]   ✅ PgBouncer is installed
[INFO]    Version: 1.24.1
2.1.12-stable
c-ares
OpenSSL
yes
[OK]   ✅ PgBouncer service is active: pgbouncer@identity-sau-main-dev.service
[OK]   ✅ PgBouncer IP service is active: pgbouncer-ip@identity-sau-main-dev.service
[OK]   ✅ PgBouncer IP: 10.100.1.204
[OK]   ✅ PgBouncer IP is bound to network interface
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini' as root on web-03.
[WARN] ⚠️ PgBouncer config not found: /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/identity-sau-main-dev/userlist.txt' as root on web-03.
[WARN] ⚠️ PgBouncer auth file not found: /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[OK]   ✅ PgBouncer is listening on port 6432

[INFO] 1️⃣2️⃣ Enhanced PgBouncer Admin Console Verification...
Failed to print table: Broken pipe
[INFO] ℹ️  PgBouncer not configured for enhanced verification

[INFO] 1️⃣3️⃣ Replicator User Connection Verification...
[INFO] Found 1 replication slot(s) - verifying replicator connectivity...
[WARN] ⚠️ Replicator certificates not found at /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[INFO]    Expected files:
[INFO]    - /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[INFO]    - /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/replicator.crt
[INFO]    - /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/replicator.key
[INFO] Checking pg_hba.conf replicator rules...
[OK]   ✅ Replicator HBA rules found:
 line_number | type | database | user_name | address | auth_method | options | error 
-------------+------+----------+-----------+---------+-------------+---------+-------
(0 rows)
[INFO] Checking active replicator connections in pg_stat_activity...
[WARN] ⚠️ No active replicator connections in pg_stat_activity
[WARN]    This is expected if standbys are not currently connected

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ PostgreSQL verification completed successfully!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Instance:       identity-sau-main-dev-coordinator
[INFO] Service:        postgresql@identity-sau-main-dev-coordinator.service
[INFO] Socket:         /var/run/postgresql-identity-sau-main-dev-coordinator
[INFO] Data Directory: /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[INFO] Hostname:       db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[INFO] Port:           5432
[INFO] SSL:            on
[INFO] WAL Level:      logical
[INFO] Citus:          yes
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 💡 OPTIMIZATION OPPORTUNITIES (Optional Enhancements)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1. Enable synchronous replication for zero-data-loss (RPO=0)
[INFO]    🛡️  Benefit: Guaranteed no data loss on primary failure (zero RPO)
[INFO]    ⚖️  Trade-off: Slightly higher write latency (~1-5ms) for durability guarantee
[INFO]    🎯 Use case: Critical data requiring absolute durability across availability zones
[INFO]    🔧 Action: ALTER SYSTEM SET synchronous_standby_names = 'ANY 1 (coordinator_standby_01, coordinator_standby_02)';
[INFO]    ⚠️  Note: Requires at least one standby to be available for writes to commit
[INFO]    📚 Docs: https://www.postgresql.org/docs/current/warm-standby.html#SYNCHRONOUS-REPLICATION

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ℹ️  These are optional enhancements for production-scale deployments
[INFO] ℹ️  Current configuration is fully functional and ready for production
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] ✅ Verification passed for coordinator


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📊 PRODUCTION READINESS CHECKS (Step 04 & 05)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔍 Checking Monitoring Setup (postgres_exporter or observability cell)...

[INFO] ℹ️  Monitoring can be configured via:
[INFO]    • Local postgres_exporter (step 04-monitoring-setup.sh)
[INFO]    • Observability Cell integration (step 02-observability-cell)

[ OK ] ✅ postgres_exporter is installed
[INFO]    Version: 0.10.1-1ubuntu0.22.04.3
[ OK ] ✅ postgres_exporter-identity-sau-main-dev.service is running
[WARN] ⚠️  Metrics endpoint not responding
[INFO] ℹ️  Monitoring user 'postgres_exporter' not found in PostgreSQL
[INFO]    This is expected if using observability cell remote monitoring
[INFO] ℹ️  Monitoring check passed (local or observability cell)

[INFO] 🔍 Checking Backup Setup (pgBackRest + WAL archiving)...

[ OK ] ✅ pgBackRest is installed
[INFO]    Version: pgBackRest 2.56.0
[ OK ] ✅ WAL archiving is enabled (archive_mode=on)
[ OK ] ✅ archive_command is configured for pgBackRest
[INFO]    Command: timeout 30 /usr/bin/pgbackrest --stanza=identity-sau-main-dev-coordinator archive-push %p
[ OK ] ✅ pgBackRest configuration exists
[ OK ] ✅ pgBackRest stanza 'identity-sau-main-dev-coordinator' is initialized
[ OK ] ✅ Backups exist (4 full backup(s))
[INFO]    Latest backup info:
                 timestamp start/stop: 2026-01-02 11:54:06+00 / 2026-01-02 11:54:16+00
                 wal start/stop: 000000010000000000000004 / 000000010000000000000004
                 database size: 37.4MB, database backup size: 37.4MB
                 repo1: backup set size: 5.7MB, backup size: 5.7MB
     
             full backup: 20260102-115435F
                 timestamp start/stop: 2026-01-02 11:54:35+00 / 2026-01-02 11:54:39+00
                 wal start/stop: 000000010000000000000007 / 000000010000000000000007
                 database size: 37.4MB, database backup size: 37.4MB
                 repo1: backup set size: 5.7MB, backup size: 5.7MB
[ OK ] ✅ Automated backup cron jobs are configured
[INFO]    Schedule:
     0 2 * * 0 root /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
     0 2 * * 1-6 root /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[ OK ] ✅ Backup directory exists: /var/lib/pgbackrest
[INFO]    Total backup size: 2.2G

[INFO] 🔍 Checking Worker Backup Coverage...

[INFO] ℹ️  Worker backups are optional for development environments
[INFO]    For production, ensure all workers have backup coverage

[INFO] Checking worker 1/1: worker-01...
[WARN] ⚠️  Worker worker-01 stanza exists but status unknown
[INFO] ℹ️  Incomplete worker backup coverage (0/1) - OK for dev

[INFO] 🔍 Checking Synchronous Replication (RPO=0)...

[INFO] ℹ️  Synchronous replication (RPO=0) is optional for development
[INFO]    For production with zero data loss requirement, enable sync replication

[INFO] ℹ️  Worker worker-01 synchronous replication NOT configured
[INFO]    └─ synchronous_commit: on
[INFO]    └─ synchronous_standby_names: 
[ OK ] ✅ All workers have synchronous replication (RPO=0)

[INFO] 🔍 Checking Connection and Memory Optimization...

[ OK ] ✅ Coordinator max_connections optimized: 150
[ OK ] ✅ Coordinator work_mem optimized: 8MB
[ OK ] ✅ Worker worker-01 max_connections optimized: 100
[ OK ] ✅ Worker worker-01 work_mem optimized: 8MB
[ OK ] ✅ All instances have optimized connection and memory settings

[INFO] 🔍 Checking Optimizations...

[ OK ] ✅ Citus coordinator host configured: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[ OK ] ✅ Periodic integrity checks configured
[INFO]    └─ Daily checks: 3, Weekly verify: 3
[WARN] ⚠️  Backup schedule NOT staggered (all at :00)
[INFO]    Optimize with: ./setup/04-postgresql/steps/04-production-optimizations.sh
[2026-01-02 12:08:27 UTC] USER=www-data EUID=0 PID=2082281 ACTION=fsop ARGS=test -f /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 12:08:27 UTC] USER=www-data EUID=0 PID=2082290 ACTION=fsop ARGS=grep -q ## Cipher Key Management /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[WARN] ⚠️  Cipher key management documentation missing
[INFO]    Add with: ./setup/04-postgresql/steps/04-production-optimizations.sh
[INFO] ℹ️  Offsite backup (repo2) not configured (optional for production)
[INFO]    Setup guide: ./setup/05-db/engine/postgresql/steps/14-setup-offsite-backup.sh
[WARN] ⚠️  Some production optimizations incomplete

[INFO] 🔍 Checking Citus Maintenance Daemon Health...

[INFO] Checking for stuck Citus Maintenance Daemons...
[ OK ] ✅ Citus Maintenance Daemons are healthy
[INFO] Checking for stuck distributed table operations...
[ OK ] ✅ No stuck distributed table operations
[INFO] Testing distributed table operations (10s timeout)...
[WARN] ⚠️  CRITICAL: Distributed table test TIMED OUT (10s)
[WARN]    Citus cluster is NOT operational - distributed tables cannot be created
[WARN]    This confirms maintenance daemons are stuck
[WARN]    
[WARN]    🔧 ACTION REQUIRED: Restart coordinator before using Citus
[WARN]       sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-coordinator.service
[INFO] Checking for lock contention...
[ OK ] ✅ No lock contention detected
[INFO] Checking for lingering prepared transactions...
[ OK ] ✅ No lingering prepared transactions

[WARN] ⚠️  Citus cluster has health issues - see warnings above
[WARN]    
[WARN]    ⚡ IMMEDIATE ACTION: Restart coordinator to restore Citus functionality
[WARN]       sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev.service

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📋 PRODUCTION READINESS SUMMARY
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Component                 Status          Production Ready?   
───────────────────────── ─────────────── ────────────────────
Citus Cluster             ✅ Operational YES                 
High Availability         ✅ Configured  YES                 
SSL/TLS Security          ✅ Enabled     YES                 
PgBouncer                 ✅ Running     YES                 
Monitoring                ✅ Operational YES                 
Backups (Coordinator)     ✅ Configured  YES                 
Backups (Workers)         ✅ Configured  YES                 
Sync Replication (RPO=0)  ✅ Enabled     YES                 
Connection Optimization   ✅ Configured  YES                 
Optimizations             ⚠️  Incomplete OPTIONAL            

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[ OK ] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] 🎉 PRODUCTION READY: 100% (3/3 critical checks passed)
[ OK ] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ All critical components are operational and production-ready

[INFO] Next steps:
[INFO]   1. Configure Prometheus to scrape metrics: http://localhost:9230/metrics
[INFO]   2. Import Grafana dashboards for PostgreSQL + Citus monitoring
[INFO]   3. Setup alerting rules for critical metrics
[INFO]   4. Schedule regular restore drills (monthly)
[INFO]   5. Review /var/www/html/skeleton.dev.fastorder.com/fixing/scripts/PRODUCTION_READINESS.md

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✓ Verification process completed successfully
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 18-production-optimizations.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PostgreSQL Production Optimizations
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Environment: identity-sau-main-dev
[INFO] Enable Sync Replication: --auto

[INFO] 1️⃣ Configuring Citus coordinator hostname...
[ OK ] ✅ Coordinator hostname already configured: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

[INFO] 2️⃣ Configuring synchronous replication for RPO=0...
[INFO] Synchronous replication NOT enabled (use './04-production-optimizations.sh yes' to enable)
[INFO] Current configuration: async replication (RPO > 0)
[INFO] 
[INFO] To enable safely after deployment:
[INFO]   /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/enable_sync_replication_safe.sh \
[INFO]     /var/run/postgresql-identity-sau-main-dev-worker-01 worker_01_standby_01

[INFO] 3️⃣ Adding periodic integrity check cron jobs...
[2026-01-02 12:08:30 UTC] USER=www-data EUID=0 PID=2082441 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-integrity-identity-sau-main-dev
[ OK ] ✅ Integrity check cron jobs configured
[INFO]    Daily checks: 02:15, 03:30, 04:45 (coordinator, worker-01, worker-02)
[INFO]    Weekly verify: Sundays at same times

[INFO] 4️⃣ Updating backup schedule with staggered timing...
[ OK ] ✅ Backup schedule staggered:
[INFO]    Coordinator: 02:05 (full: Sun, diff: Mon-Sat)
[INFO]    Worker-01:   03:10 (full: Sun, diff: Mon-Sat)
[INFO]    Worker-02:   04:15 (full: Sun, diff: Mon-Sat)

[INFO] 5️⃣ Documenting cipher key backup procedures...
[2026-01-02 12:08:30 UTC] USER=www-data EUID=0 PID=2082468 ACTION=fsop ARGS=test -f /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
Sorry, user www-data is not allowed to execute '/usr/bin/grep -q ## Cipher Key Management /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md' as root on web-03.
[ OK ] ✅ Cipher key documentation added to /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO] 6️⃣ Checking offsite backup configuration...
[INFO] ℹ️  Offsite backup (repo2) is NOT configured
[INFO]    Configuration example: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[INFO]    Setup instructions: ./setup/04-postgresql/steps/08-setup-offsite-backup.sh
[ OK ] ✅ Offsite backup example available: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Production Optimizations Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[ OK ] Applied optimizations:
[ OK ]   1. ✅ Citus coordinator hostname
[ OK ]   2. ⏭️  Synchronous replication (RPO=0)
[ OK ]   3. ✅ Periodic integrity checks (daily + weekly)
[ OK ]   4. ✅ Staggered backup schedule (reduced load spikes)
[ OK ]   5. ✅ Cipher key backup documentation
[ OK ]   6. ✅ Offsite backup (repo2) example configuration

[INFO] Next steps:
[INFO]   1. Backup cipher keys to secure vault immediately
[INFO]   2. Set up S3/MinIO for offsite backups:
[INFO]      - Instructions: ./setup/04-postgresql/steps/08-setup-offsite-backup.sh
[INFO]      - Example config: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[INFO]   3. Configure alerting for backup failures (cron exit codes)
[INFO]   4. Test restore drill from offsite repository
[INFO]   5. Enable RPO=0 if needed: ./04-production-optimizations.sh yes

[ OK ] System is now production-grade! 🎉

✓ ✔ PostgreSQL creation completed
[INFO] Cleaning up temporary files...
[INFO] Starting cleanup of temporary files...
[INFO] Cleaning up SSL temp files for identity-sau-main-dev...
[INFO] Cleaning up old provisioning logs...
[SUCCESS] Removed 9 old log files
[INFO] Cleaning up old configuration backups...
✓ ✔ Cleanup completed

✓ ✅ Database infrastructure (postgresql) setup completed successfully

[INFO] Loaded from topology.json: identity-sau-main-dev
[2026-01-02 12:08:33] Loaded environment: identity-sau-main-dev
[2026-01-02 12:08:33] Service: identity, Zone: sau, Branch: main, Env: dev
[2026-01-02 12:08:33] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-02 12:08:33] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-02 12:08:33] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
[INFO] Starting finalizing setup process...
[INFO] Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps
[INFO] Environment: identity-sau-main-dev

[INFO] Found 3 step(s) to execute

[INFO] 📦 Step 1/3: enable_disable_all_applications...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
════════════════════════════════════════════════════════════════════════════════
  Environment Services Management
════════════════════════════════════════════════════════════════════════════════
  Environment:  identity-sau-main-dev
  Action:       enable
  Triggered by: false
════════════════════════════════════════════════════════════════════════════════

🔍 Scanning for environment-specific services...
✅ Found 8 services for environment: identity-sau-main-dev

📋 Services to enable:
────────────────────────────────────────────────────────────────────────────────
  • confluent-connect-identity-sau-main-dev_coordinator.service  [active/unmasked/enabled]
  • confluent-kraft-identity-sau-main-dev_coordinator.service    [active/unmasked/enabled]
  • elasticsearch@identity-sau-main-dev-node-01.service          [active/unmasked/enabled]
  • pgbouncer-ip@identity-sau-main-dev.service                   [active/unmasked/enabled]
  • pgbouncer@identity-sau-main-dev.service                      [active/unmasked/enabled]
  • postgresql@identity-sau-main-dev-coordinator.service         [active/unmasked/enabled]
  • postgresql@identity-sau-main-dev-worker-01-standby-01.service [active/unmasked/enabled]
  • postgresql@identity-sau-main-dev-worker-01.service           [active/unmasked/enabled]
────────────────────────────────────────────────────────────────────────────────


❌ Cancelled by user
[OK] ✅ Step 1 completed: 01-enable_disable_all_applications.sh

[INFO] 📦 Step 2/3: verify monitoring...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Monitoring Verification for identity-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Detecting installed services...
Failed to print table: Broken pipe
[OK] ✓ PostgreSQL detected
Failed to print table: Broken pipe
[OK] ✓ Elasticsearch detected
Failed to print table: Broken pipe
[OK] ✓ Kafka detected
Failed to print table: Broken pipe
[OK] ✓ PgBouncer detected

[INFO] Services to verify: postgresql elasticsearch kafka pgbouncer

[INFO] 2️⃣ Verifying exporters are running...
[OK] ✓ PostgreSQL exporter is running
[OK] ✓ Elasticsearch exporter is running
[OK] ✓ Kafka JMX exporter is running
[WARN] ⚠️  PgBouncer exporter is not running (may not be configured)

[INFO] 3️⃣ Verifying Prometheus configuration...
[2026-01-02 12:08:35 UTC] USER=www-data EUID=0 PID=2082923 ACTION=passthru ARGS=grep -q job_name: 'postgresql' /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[OK] ✓ postgresql is configured in Prometheus
[2026-01-02 12:08:35 UTC] USER=www-data EUID=0 PID=2082944 ACTION=passthru ARGS=grep -q job_name: 'elasticsearch' /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[OK] ✓ elasticsearch is configured in Prometheus
[2026-01-02 12:08:35 UTC] USER=www-data EUID=0 PID=2082965 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[WARN] ⚠️  kafka is not configured in Prometheus scrape targets
[2026-01-02 12:08:36 UTC] USER=www-data EUID=0 PID=2082987 ACTION=passthru ARGS=grep -q job_name: 'pgbouncer' /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[WARN] ⚠️  pgbouncer is not configured in Prometheus scrape targets

[INFO] 4️⃣ Verifying Prometheus is actively scraping...
[OK] ✓ Prometheus is running
[OK] ✓ postgresql target is UP in Prometheus
[OK] ✓ elasticsearch target is UP in Prometheus
[WARN] ⚠️  kafka target is not UP in Prometheus (may still be initializing)
[WARN] ⚠️  pgbouncer target is not UP in Prometheus (may still be initializing)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Monitoring Verification Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[WARN] Some monitoring issues were detected:

[WARN] Prometheus Configuration Issues:
  - kafka not configured in Prometheus
  - pgbouncer not configured in Prometheus

[WARN] Automatically running monitoring setup scripts to fix issues...

[INFO] Running Kafka monitoring setup...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Kafka Monitoring Integration for identity-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[OK]   ✓ Observability cell is ready

[INFO] 2️⃣ Setting up Kafka JMX exporter integration...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[INFO] Setting up Kafka JMX exporter for identity-sau-main-dev
[INFO] JMX Prometheus Java Agent already exists at /opt/kafka/libs/jmx_prometheus_javaagent.jar
[2026-01-02 12:08:42 UTC] USER=www-data EUID=0 PID=2083088 ACTION=passthru ARGS=mv /tmp/jmx_exporter.yml /opt/kafka/config/jmx_exporter.yml
[2026-01-02 12:08:42 UTC] USER=www-data EUID=0 PID=2083097 ACTION=passthru ARGS=chmod 644 /opt/kafka/config/jmx_exporter.yml
[OK]   JMX exporter configuration created at /opt/kafka/config/jmx_exporter.yml
[OK]   JMX exporter configuration created
[INFO] Configuring Kafka systemd services to use JMX exporter...
[2026-01-02 12:08:42 UTC] USER=www-data EUID=0 PID=2083121 ACTION=fsop ARGS=test -f /etc/systemd/system/[2026-01-02
[INFO] All Kafka services already configured with JMX exporter
[OK]   Kafka JMX exporter integration complete
[INFO] Metrics endpoint: http://142.93.238.16:9308/metrics
[INFO] Prometheus will automatically scrape: https://metrics-identity-sau-main-dev.fastorder.com:9090
[INFO] View dashboards at: https://dashboards-identity-sau-main-dev.fastorder.com
[OK]   ✓ Kafka JMX exporter integration complete
[INFO] Configuring KAFKA_OPTS environment variable for kafka user...
[2026-01-02 12:08:42 UTC] USER=www-data EUID=0 PID=2083142 ACTION=passthru ARGS=grep -q KAFKA_OPTS.*javaagent.*jmx_prometheus_javaagent /home/kafka/.bashrc
[OK]   ✓ KAFKA_OPTS already configured
[INFO] 2.5️⃣ Enabling JMX exporter in Kafka systemd service...
[2026-01-02 12:08:42 UTC] USER=www-data EUID=0 PID=2083167 ACTION=passthru ARGS=grep -q javaagent.*jmx_prometheus_javaagent /etc/systemd/system/confluent-kraft-identity-sau-main-dev_coordinator.service
[OK]   ✓ JMX exporter already enabled in Kafka systemd services
[INFO] 2.6️⃣ Configuring Prometheus to scrape Kafka metrics...
[2026-01-02 12:08:42 UTC] USER=www-data EUID=0 PID=2083188 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[INFO] Adding Kafka scrape target to Prometheus configuration...
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[2026-01-02 12:08:42 UTC] USER=www-data EUID=0 PID=2083221 ACTION=passthru ARGS=sed -i /# Prometheus self-monitoring/r /tmp/prometheus_kafka_add.yml /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[ERROR] Invalid Prometheus configuration - rolling back
[2026-01-02 12:08:42 UTC] USER=www-data EUID=0 PID=2083259 ACTION=passthru ARGS=sed -i /job_name: 'kafka'/,+6d /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[2026-01-02 12:08:42 UTC] USER=www-data EUID=0 PID=2083381 ACTION=fsop ARGS=rm -f /tmp/prometheus_kafka_add.yml

[INFO] 3️⃣ Registering Kafka nodes to monitoring database...
[INFO] Registering Kafka Broker to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Kafka Broker
[INFO]   Identifier:        identity-sau-main-dev-broker-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                142.93.238.16
[INFO]   Port:              9092
[INFO]   FQDN:              eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 1a310579-24b9-4091-8626-7335f80305c3
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ Kafka broker registered
[INFO] Registering Kafka Connect to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Kafka Connect
[INFO]   Identifier:        identity-sau-main-dev-connect-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8083
[INFO]   FQDN:              eventbus-identity-sau-main-dev-kafka-connect.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 71719f62-65ea-4a2b-a0ed-4a8d3f80403b
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ Kafka Connect registered
[INFO] Schema Registry not running, skipping registration

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Kafka Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Metrics: http://localhost:9308/metrics
[INFO] Prometheus: https://metrics-identity-sau-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-identity-sau-main-dev.fastorder.com
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✓ Kafka monitoring setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Step 2 completed: 02-verify-monitoring.sh

[INFO] 📦 Step 3/3: register backup infrastructure...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔧 Registering Core Services & Backup Infrastructure for identity-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Registering Main App...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Main App
[INFO]   Identifier:        identity-sau-main-dev-main-app
[INFO]   Identifier Parent: application
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8080
[INFO]   FQDN:              app-identity-sau-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 6781a0e7-58d6-4224-ae59-10b617367a2a
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
/opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps/03-register-backup-infrastructure.sh: line 70: ok: command not found

[INFO] 2️⃣ Registering Audit Service...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Audit Service
[INFO]   Identifier:        identity-sau-main-dev-audit
[INFO]   Identifier Parent: application
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8081
[INFO]   FQDN:              audit-identity-sau-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 203de866-853b-49eb-80a0-dffa65ac5d16
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
/opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps/03-register-backup-infrastructure.sh: line 85: ok: command not found

[INFO] 3️⃣ Registering PostgreSQL Backup Node...
[ERROR] Invalid identifier format: backup-db
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register PostgreSQL backup node (non-blocking)

[INFO] 4️⃣ Registering Elasticsearch Backup Node...
[ERROR] Invalid identifier format: backup-search
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Elasticsearch backup node (non-blocking)

[INFO] 5️⃣ Registering Kafka Backup Node...
[ERROR] Invalid identifier format: backup-eventbus
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Kafka backup node (non-blocking)

[INFO] 6️⃣ Registering Backup Orchestrator...
[ERROR] Invalid identifier format: backup-orchestrator
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Backup orchestrator (non-blocking)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Core Services & Backup Infrastructure Registration Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Registered core services:
[INFO]   🚀 main-app            → Core application service
[INFO]   📋 audit               → Centralized audit logging (WORM)

[INFO] Registered backup nodes:
[INFO]   📦 backup-db           → PostgreSQL backup (pgBackRest, PITR)
[INFO]   📦 backup-search       → Elasticsearch snapshots (ILM, S3)
[INFO]   📦 backup-eventbus     → Kafka log segments (replication)
[INFO]   📦 backup-orchestrator → Central backup coordination

[INFO] Dashboard: https://skeleton.dev.fastorder.com/dashboard/monitoring
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Step 3 completed: 03-register-backup-infrastructure.sh


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ finalizing setup completed successfully!
[OK] Executed all 3 steps
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Environment: identity-sau-main-dev
[INFO] Service: identity
[INFO] Zone: sau
[INFO] Branch: main
[INFO] Env: dev