📊 Provisioning Job Status

[INFO] Using database engine from DB_ENGINE environment variable: postgresql
[INFO] Cleaning up any existing locks...

Starting database engine: postgresql
═══════════════════════════════════════════════

[INFO] Loaded from topology.json: identity-sau-main-dev
[2026-01-02 08:10:00] Loaded environment: identity-sau-main-dev
[2026-01-02 08:10:00] Service: identity, Zone: sau, Branch: main, Env: dev
[2026-01-02 08:10:00] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-02 08:10:00] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-02 08:10:00] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[INFO] Observability cell verified for identity-sau-main-dev
[INFO] Monitoring will be configured after PostgreSQL deployment (step 10-monitoring-setup.sh)
[INFO] Citus mode ENABLED
[INFO] → Coordinator + 1 worker(s) + 1 standby node(s) per worker
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up coordinator (Citus control plane)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-02 08:10:02 UTC] USER=unknown EUID=33 PID=1707363 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 08:10:02 UTC] USER=unknown EUID=33 PID=1707370 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 08:10:02 UTC] USER=unknown EUID=33 PID=1707377 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-02 08:10:02 UTC] USER=unknown EUID=33 PID=1707384 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 08:10:02 UTC] USER=unknown EUID=33 PID=1707391 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-02 08:10:02 UTC] USER=unknown EUID=33 PID=1707398 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4 JOB_UUID=0c5008d4-812c-463b-922c-ff476c8d9257

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] PostgreSQL IP: 10.100.1.213
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

[INFO] Adding /etc/hosts entries for coordinator...
[INFO]   1. db-identity-sau-main-dev-postgresql.fastorder.com → 10.100.1.213 (primary/short)
[INFO]   2. db-identity-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.213 (compatibility)

[INFO]   ✅ db-identity-sau-main-dev-postgresql.fastorder.com already exists with correct IP
[INFO]   ✅ db-identity-sau-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.213    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  10.100.1.213    db-identity-sau-main-dev-postgresql.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 08:10:07 UTC] USER=www-data EUID=0 PID=1707643 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator and /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:10:07 UTC] USER=www-data EUID=0 PID=1707652 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-02 08:10:07 UTC] USER=www-data EUID=0 PID=1707662 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1707610
[2026-01-02 08:10:07 UTC] USER=www-data EUID=0 PID=1707671 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1707610/ra_root.crt
[2026-01-02 08:10:07 UTC] USER=www-data EUID=0 PID=1707680 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1707610/ra_root.key
[2026-01-02 08:10:07 UTC] USER=www-data EUID=0 PID=1707689 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1707610/ra_root.crt
[2026-01-02 08:10:07 UTC] USER=www-data EUID=0 PID=1707699 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1707610/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-02 08:10:10 UTC] USER=www-data EUID=0 PID=1707739 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1707610/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 08:10:10 UTC] USER=www-data EUID=0 PID=1707748 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1707610/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 08:10:10 UTC] USER=www-data EUID=0 PID=1707757 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-02 08:10:10 UTC] USER=www-data EUID=0 PID=1707766 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1707610/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:10 UTC] USER=www-data EUID=0 PID=1707775 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:10 UTC] USER=www-data EUID=0 PID=1707784 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:10 UTC] USER=www-data EUID=0 PID=1707793 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 08:10:10 UTC] USER=www-data EUID=0 PID=1707804 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 08:10:10 UTC] USER=www-data EUID=0 PID=1707813 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 08:10:10 UTC] USER=www-data EUID=0 PID=1707822 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 08:10:10 UTC] USER=www-data EUID=0 PID=1707831 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 08:10:10 UTC] USER=www-data EUID=0 PID=1707840 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:10:10 UTC] USER=www-data EUID=0 PID=1707849 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        coordinator
Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:10:11 UTC] USER=www-data EUID=0 PID=1707906 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-01-02 08:10:11 UTC] USER=www-data EUID=0 PID=1707915 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 08:10:11 UTC] USER=www-data EUID=0 PID=1707924 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-01-02 08:10:11 UTC] USER=www-data EUID=0 PID=1707933 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 08:10:11 UTC] USER=www-data EUID=0 PID=1707942 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1707956 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1707965 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1707974 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1707983 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1707992 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1708001 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1708010 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1708019 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1708028 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1708037 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1708046 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1708055 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1708064 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1708073 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1708082 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1708091 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1708117 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1708126 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1708135 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:10:12 UTC] USER=www-data EUID=0 PID=1708144 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708153 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708162 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708171 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708180 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708189 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708198 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708207 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708217 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708227 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708236 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708245 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708254 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708263 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708272 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708281 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708290 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708299 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708308 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708317 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708327 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708337 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708346 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708355 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708364 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:13 UTC] USER=www-data EUID=0 PID=1708373 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708382 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708391 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708400 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708409 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708418 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708427 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708437 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708449 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708458 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708467 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708476 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708485 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708494 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708503 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708512 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708521 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708530 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708539 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708549 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708559 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708568 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708577 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708586 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708595 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708604 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708613 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708622 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:10:14 UTC] USER=www-data EUID=0 PID=1708631 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:10:15 UTC] USER=www-data EUID=0 PID=1708672 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-01-02 08:10:15 UTC] USER=www-data EUID=0 PID=1708681 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 08:10:15 UTC] USER=www-data EUID=0 PID=1708690 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-01-02 08:10:15 UTC] USER=www-data EUID=0 PID=1708699 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 08:10:15 UTC] USER=www-data EUID=0 PID=1708708 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708722 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708733 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708742 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708751 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708760 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708769 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708778 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708787 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708796 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708805 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708814 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708823 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708832 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708841 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708850 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708859 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708868 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708877 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708903 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708912 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708921 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708930 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708939 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708948 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708957 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708966 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708975 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:10:16 UTC] USER=www-data EUID=0 PID=1708984 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1708993 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709003 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709013 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709022 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709031 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709040 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709049 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709058 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709067 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709076 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709085 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709094 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709103 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709113 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709123 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709132 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709141 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709150 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709159 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709168 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709177 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709186 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709195 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709204 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709213 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709223 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709233 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709242 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709251 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709260 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:17 UTC] USER=www-data EUID=0 PID=1709269 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:10:18 UTC] USER=www-data EUID=0 PID=1709278 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:10:18 UTC] USER=www-data EUID=0 PID=1709287 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:10:18 UTC] USER=www-data EUID=0 PID=1709296 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:18 UTC] USER=www-data EUID=0 PID=1709305 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:10:18 UTC] USER=www-data EUID=0 PID=1709315 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:10:18 UTC] USER=www-data EUID=0 PID=1709326 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:10:18 UTC] USER=www-data EUID=0 PID=1709336 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:10:18 UTC] USER=www-data EUID=0 PID=1709346 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:10:18 UTC] USER=www-data EUID=0 PID=1709355 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:10:18 UTC] USER=www-data EUID=0 PID=1709364 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:10:18 UTC] USER=www-data EUID=0 PID=1709382 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:10:18 UTC] USER=www-data EUID=0 PID=1709391 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:10:18 UTC] USER=www-data EUID=0 PID=1709400 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:10:18 UTC] USER=www-data EUID=0 PID=1709409 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:10:18 UTC] USER=www-data EUID=0 PID=1709420 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-coordinator-postgresql environment: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com (10.100.1.213)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.213
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] Data dir:   /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[INFO] Port:       5432
[INFO] Hostname:   db-identity-sau-main-dev-postgresql-coordinator
[2026-01-02 08:10:20 UTC] USER=www-data EUID=0 PID=1709517 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:10:20 UTC] USER=www-data EUID=0 PID=1709538 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:10:21 UTC] USER=www-data EUID=0 PID=1709561 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:10:21 UTC] USER=www-data EUID=0 PID=1709582 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 08:10:21 UTC] USER=www-data EUID=0 PID=1709622 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator and /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:10:21 UTC] USER=www-data EUID=0 PID=1709631 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-02 08:10:21 UTC] USER=www-data EUID=0 PID=1709641 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1709589
[2026-01-02 08:10:21 UTC] USER=www-data EUID=0 PID=1709650 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1709589/ra_root.crt
[2026-01-02 08:10:21 UTC] USER=www-data EUID=0 PID=1709659 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1709589/ra_root.key
[2026-01-02 08:10:22 UTC] USER=www-data EUID=0 PID=1709668 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1709589/ra_root.crt
[2026-01-02 08:10:22 UTC] USER=www-data EUID=0 PID=1709677 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1709589/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-02 08:10:23 UTC] USER=www-data EUID=0 PID=1709713 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1709589/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 08:10:23 UTC] USER=www-data EUID=0 PID=1709722 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1709589/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 08:10:23 UTC] USER=www-data EUID=0 PID=1709731 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-02 08:10:23 UTC] USER=www-data EUID=0 PID=1709740 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1709589/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:24 UTC] USER=www-data EUID=0 PID=1709749 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:24 UTC] USER=www-data EUID=0 PID=1709758 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:10:24 UTC] USER=www-data EUID=0 PID=1709767 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 08:10:24 UTC] USER=www-data EUID=0 PID=1709778 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 08:10:24 UTC] USER=www-data EUID=0 PID=1709787 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 08:10:24 UTC] USER=www-data EUID=0 PID=1709796 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 08:10:24 UTC] USER=www-data EUID=0 PID=1709805 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 08:10:24 UTC] USER=www-data EUID=0 PID=1709816 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:10:24 UTC] USER=www-data EUID=0 PID=1709825 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        coordinator
Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-02 08:10:24 UTC] USER=www-data EUID=0 PID=1709854 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 08:10:24 UTC] USER=www-data EUID=0 PID=1709863 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 08:10:24 UTC] USER=www-data EUID=0 PID=1709872 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-02 08:10:24 UTC] USER=www-data EUID=0 PID=1709893 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 08:10:24 UTC] USER=www-data EUID=0 PID=1709917 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-02 08:10:25 UTC] USER=www-data EUID=0 PID=1709948 ACTION=fsop ARGS=rm -rf /var/run/postgresql-identity-sau-main-dev-coordinator
[OK]   No conflicting Postgres left on port 5432
[OK]   Using postgres password from vault provider
[2026-01-02 08:10:28 UTC] USER=www-data EUID=0 PID=1710003 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.Vmdam3
[2026-01-02 08:10:28 UTC] USER=www-data EUID=0 PID=1710024 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.Vmdam3
[2026-01-02 08:10:28 UTC] USER=www-data EUID=0 PID=1710046 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 08:10:28 UTC] USER=www-data EUID=0 PID=1710068 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 08:10:28 UTC] USER=www-data EUID=0 PID=1710090 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/identity-sau-main-dev
[INFO] Initializing cluster in /var/lib/postgresql/17/identity-sau-main-dev/coordinator (SCRAM; pwfile)
[WARN] Removing existing data directory: /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 08:10:28 UTC] USER=www-data EUID=0 PID=1710111 ACTION=fsop ARGS=rm -rf /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 08:10:29 UTC] USER=www-data EUID=0 PID=1710134 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 08:10:29 UTC] USER=www-data EUID=0 PID=1710189 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 08:10:29 UTC] USER=www-data EUID=0 PID=1710275 ACTION=fsop ARGS=chmod 700 /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 08:10:29 UTC] USER=www-data EUID=0 PID=1710296 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-02 08:10:29 UTC] USER=www-data EUID=0 PID=1710318 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-02 08:10:29 UTC] USER=www-data EUID=0 PID=1710339 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-02 08:10:29 UTC] USER=www-data EUID=0 PID=1710348 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /var/lib/postgresql/17/identity-sau-main-dev/coordinator --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.Vmdam3
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/17/identity-sau-main-dev/coordinator ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /var/lib/postgresql/17/identity-sau-main-dev/coordinator -l logfile start

[OK]   initdb complete
[2026-01-02 08:10:30 UTC] USER=www-data EUID=0 PID=1710383 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.Vmdam3
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-02 08:10:31 UTC] USER=www-data EUID=0 PID=1710432 ACTION=fsop ARGS=cp /tmp/tmp.9dAkX1QQ6v /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
[2026-01-02 08:10:31 UTC] USER=www-data EUID=0 PID=1710453 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
[2026-01-02 08:10:31 UTC] USER=www-data EUID=0 PID=1710474 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 08:10:31 UTC] USER=www-data EUID=0 PID=1710499 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.F5lCKS /etc/systemd/system/postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 08:10:31 UTC] USER=www-data EUID=0 PID=1710522 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@identity-sau-main-dev-coordinator.service
[OK]   systemd unit written
[2026-01-02 08:10:31 UTC] USER=www-data EUID=0 PID=1710544 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 08:10:31 UTC] USER=www-data EUID=0 PID=1710566 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 08:10:31 UTC] USER=www-data EUID=0 PID=1710587 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-02 08:10:35 UTC] USER=www-data EUID=0 PID=1710708 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-02 08:10:36 UTC] USER=www-data EUID=0 PID=1710750 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Provisioning application database and Debezium role (if not exists)...
[INFO] Checking if database fastorder_identity_sau_main_dev_db exists...
[INFO] DB check result: exit_code=0, output='[2026-01-02 08:10:37 UTC] USER=www-data EUID=0 PID=1710907 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_identity_sau_main_dev_db''
[INFO] Creating database fastorder_identity_sau_main_dev_db...
[2026-01-02 08:10:37 UTC] USER=www-data EUID=0 PID=1710930 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_identity_sau_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[OK]   Database fastorder_identity_sau_main_dev_db created
[INFO] Checking if role debezium_user exists...
[INFO] Role check result: exit_code=0, output='[2026-01-02 08:10:38 UTC] USER=www-data EUID=0 PID=1710955 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[INFO] Creating role debezium_user...
[2026-01-02 08:10:38 UTC] USER=www-data EUID=0 PID=1710982 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'zBfRMgTdK99vjsm17XxPVzOR';
CREATE ROLE
[OK]   Role debezium_user created
[2026-01-02 08:10:38 UTC] USER=www-data EUID=0 PID=1711006 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_identity_sau_main_dev_db" TO debezium_user;
GRANT
[OK]   Application DB (fastorder_identity_sau_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[INFO] Applying connection and memory optimizations...
[INFO] Current settings: max_connections=100, work_mem=4MB
[INFO] Target settings (coordinator): max_connections=150, work_mem=8MB
[2026-01-02 08:10:39 UTC] USER=www-data EUID=0 PID=1711092 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 150;
ALTER SYSTEM
[2026-01-02 08:10:39 UTC] USER=www-data EUID=0 PID=1711116 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-02 08:10:39 UTC] USER=www-data EUID=0 PID=1711139 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   Settings applied to postgresql.auto.conf
[2026-01-02 08:10:39 UTC] USER=www-data EUID=0 PID=1711154 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/standby.signal
[INFO] Service recently started (3s ago) - restarting to apply max_connections...
[INFO] Stopping service...
[2026-01-02 08:10:39 UTC] USER=www-data EUID=0 PID=1711177 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[INFO] Waiting for port 5432 to be released...
[OK]   Port 5432 released
[INFO] Starting service...
[2026-01-02 08:10:43 UTC] USER=www-data EUID=0 PID=1711246 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 08:10:49 UTC] USER=www-data EUID=0 PID=1711294 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[OK]   ✅ Optimization complete: max_connections=150, work_mem=8MB
[INFO] Setting postgres password via centralized script... for coordinator
[INFO] Temporarily disabling synchronous_commit on coordinator for password setting...
[OK]   Disabled synchronous_commit (was: on)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Initial setup: Using password from initdb
✓ PostgreSQL password already set during initdb
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/coordinator
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
[INFO] Restoring synchronous_commit on coordinator...
[OK]   Restored synchronous_commit to: on
[OK]   Password set and persisted
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] PostgreSQL IP: 10.100.1.213
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

[INFO] Adding /etc/hosts entries for coordinator...
[INFO]   1. db-identity-sau-main-dev-postgresql.fastorder.com → 10.100.1.213 (primary/short)
[INFO]   2. db-identity-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.213 (compatibility)

[INFO]   ✅ db-identity-sau-main-dev-postgresql.fastorder.com already exists with correct IP
[INFO]   ✅ db-identity-sau-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.213    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  10.100.1.213    db-identity-sau-main-dev-postgresql.fastorder.com


[OK]   PostgreSQL 'identity-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key \
        host=db-identity-sau-main-dev-postgresql-coordinator port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.213
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-coordinator
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: ce097707-5ce5-40c8-a941-01512555cab8
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 📦 03 role...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 08:11:04 UTC] USER=www-data EUID=0 PID=1711947 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    debezium_user
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   debezium_user
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:11:28 UTC] USER=www-data EUID=0 PID=1712163 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-debezium_user
[2026-01-02 08:11:28 UTC] USER=www-data EUID=0 PID=1712172 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-01-02 08:11:28 UTC] USER=www-data EUID=0 PID=1712181 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-debezium_user/ra_root.key
[2026-01-02 08:11:28 UTC] USER=www-data EUID=0 PID=1712190 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-01-02 08:11:28 UTC] USER=www-data EUID=0 PID=1712199 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:11:28 UTC] USER=www-data EUID=0 PID=1712213 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:11:28 UTC] USER=www-data EUID=0 PID=1712222 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:11:28 UTC] USER=www-data EUID=0 PID=1712231 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 08:11:28 UTC] USER=www-data EUID=0 PID=1712240 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 08:11:28 UTC] USER=www-data EUID=0 PID=1712249 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:11:28 UTC] USER=www-data EUID=0 PID=1712258 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:11:28 UTC] USER=www-data EUID=0 PID=1712267 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 08:11:28 UTC] USER=www-data EUID=0 PID=1712276 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 08:11:28 UTC] USER=www-data EUID=0 PID=1712287 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712296 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712305 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712318 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712383 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712418 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712448 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712478 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712489 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712498 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712507 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712516 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712525 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712535 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712545 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712554 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712563 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712572 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712582 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712592 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712601 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712610 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712619 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712628 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:29 UTC] USER=www-data EUID=0 PID=1712638 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712647 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712656 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712665 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712674 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712683 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712693 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712703 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712712 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712721 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712730 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712739 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712749 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712758 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712767 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712776 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712785 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712794 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712804 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712814 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712823 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712832 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712841 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712850 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712859 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712868 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712877 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712887 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712896 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 08:11:30 UTC] USER=www-data EUID=0 PID=1712907 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 08:11:31 UTC] USER=www-data EUID=0 PID=1712917 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:11:31 UTC] USER=www-data EUID=0 PID=1712929 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:11:31 UTC] USER=www-data EUID=0 PID=1712938 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:11:31 UTC] USER=www-data EUID=0 PID=1712947 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:11:31 UTC] USER=www-data EUID=0 PID=1712956 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:11:31 UTC] USER=www-data EUID=0 PID=1712965 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:11:31 UTC] USER=www-data EUID=0 PID=1712974 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:11:31 UTC] USER=www-data EUID=0 PID=1712983 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:11:31 UTC] USER=www-data EUID=0 PID=1712993 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:11:31 UTC] USER=www-data EUID=0 PID=1713003 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: debezium_user
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U debezium_user -d postgres

✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: fastorder_admin_gd
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: coordinator/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-identity-sau-main-dev-postgresql.fastorder.com
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    fastorder_admin_gd
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   fastorder_admin_gd
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:11:47 UTC] USER=www-data EUID=0 PID=1713465 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-fastorder_admin_gd
[2026-01-02 08:11:47 UTC] USER=www-data EUID=0 PID=1713474 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-02 08:11:47 UTC] USER=www-data EUID=0 PID=1713483 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
[2026-01-02 08:11:47 UTC] USER=www-data EUID=0 PID=1713492 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-02 08:11:47 UTC] USER=www-data EUID=0 PID=1713501 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713523 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713533 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713542 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713551 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713560 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713569 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713578 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713587 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713596 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713605 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713614 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713625 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713634 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713643 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713652 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713661 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713670 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713679 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:48 UTC] USER=www-data EUID=0 PID=1713706 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713715 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713724 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713733 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713744 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713753 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713762 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713771 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713780 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713791 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713801 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713811 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713821 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713830 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713839 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713848 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713857 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713866 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713875 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713884 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713893 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713902 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713912 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713922 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713932 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713942 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713962 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1713995 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1714028 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:49 UTC] USER=www-data EUID=0 PID=1714060 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714085 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714094 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714104 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714113 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714122 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714132 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714142 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714151 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714160 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714170 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714179 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714188 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714197 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714206 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714215 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714224 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714233 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714243 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714253 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714264 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714273 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714282 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714291 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714300 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714309 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714318 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:11:50 UTC] USER=www-data EUID=0 PID=1714327 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: fastorder_admin_gd
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-identity-sau-main-dev-coordinator:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_identity_sau_main_dev_db already exists, skipping creation
[2026-01-02 08:11:51 UTC] USER=www-data EUID=0 PID=1714389 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-02 08:11:51 UTC] USER=www-data EUID=0 PID=1714423 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

=== Pre-flight Checks ===
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
✓ AWS Secrets Manager accessible

=== Retrieving Credentials from AWS ===
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
/opt/fastorder/bash/infra_core/cache.sh: line 145: /var/cache/secrets/fastorder_db_identity_sau_main_dev_postgresql_coordinator_fastorder_admin_gd.cache.tmp.1714443: Permission denied
✅ Retrieved from secrets manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials loaded for coordinator/fastorder_admin_gd: fastorder_admin_gd@db-identity-sau-main-dev-postgresql.fastorder.com:5432/fastorder_identity_sau_main_dev_db
✓ Credentials retrieved: fastorder_admin_gd@db-identity-sau-main-dev-postgresql.fastorder.com:5432/fastorder_identity_sau_main_dev_db
╔════════════════════════════════════════════╗
║  PostgreSQL Test Suite (AWS Secrets MGR)  ║
╚════════════════════════════════════════════╝

=== PostgreSQL Authentication Test ===
✗ PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-identity-sau-main-dev-postgresql.fastorder.com" (10.100.1.213), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 08:12:00 UTC] USER=www-data EUID=0 PID=1714647 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : coordinator
  PG HOST     : db-identity-sau-main-dev-postgresql.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_identity_sau_main_dev_db
  SCHEMA      : auth
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
  DNS → 10.100.1.213
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User debezium_user does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: debezium_user
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: coordinator/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-01-02 08:12:15 UTC] USER=www-data EUID=0 PID=1715164 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
[2026-01-02 08:12:15 UTC] USER=www-data EUID=0 PID=1715190 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d fastorder_identity_sau_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=coordinator IDENTIFIER_PARENT=coordinator
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║                    IAM Database Schema Initialization                       ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] 🟢 Starting IAM schema provisioning...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

[INFO] 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: core
  Core Identity Directory (tenants, realms, identities, devices, MFA)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [1/20]: core/01-tenant
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
🔧 Installing extensions...
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
   Creating reference table: core.tenant
 create_reference_table 
------------------------
 
(1 row)

✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[OK] Table core/01-tenant initialized

[INFO] 🔸 Table [2/20]: core/02-realm
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.realm$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_realm_updated" for relation "core.realm" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[OK] Table core/02-realm initialized

[INFO] 🔸 Table [3/20]: core/03-identity
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_identity_updated" for relation "core.identity" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[OK] Table core/03-identity initialized

[INFO] 🔸 Table [4/20]: core/04-device
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.device$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[OK] Table core/04-device initialized

[INFO] 🔸 Table [5/20]: core/05-identity_account
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity_account$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_identity_account_updated" for relation "core.identity_account" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[OK] Table core/05-identity_account initialized

[INFO] 🔸 Table [6/20]: core/06-identity_mfa
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity_mfa$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[OK] Table core/06-identity_mfa initialized

[INFO] 🔸 Table [7/20]: core/07-external_idp_link
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.external_idp_link$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[OK] Table core/07-external_idp_link initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: policy
  RBAC/ABAC Authorization (clients, roles, permissions, policies)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [8/20]: policy/01-client
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.client$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
NOTICE:  trigger "tr_client_updated" for relation "policy.client" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[OK] Table policy/01-client initialized

[INFO] 🔸 Table [9/20]: policy/02-resource
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.resource$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[OK] Table policy/02-resource initialized

[INFO] 🔸 Table [10/20]: policy/03-scope
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.scope$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_scope_updated" for relation "policy.scope" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[OK] Table policy/03-scope initialized

[INFO] 🔸 Table [11/20]: policy/04-permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.permission$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_permission_updated" for relation "policy.permission" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[OK] Table policy/04-permission initialized

[INFO] 🔸 Table [12/20]: policy/05-role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.role$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_role_updated" for relation "policy.role" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[OK] Table policy/05-role initialized

[INFO] 🔸 Table [13/20]: policy/06-role_permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.role_permission$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[OK] Table policy/06-role_permission initialized

[INFO] 🔸 Table [14/20]: policy/07-identity_role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.identity_role$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[OK] Table policy/07-identity_role initialized

[INFO] 🔸 Table [15/20]: policy/08-policy_rule
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.policy_rule$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_policy_rule_updated" for relation "policy.policy_rule" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[OK] Table policy/08-policy_rule initialized

[INFO] 🔸 Table [16/20]: policy/09-api_key
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.api_key$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[OK] Table policy/09-api_key initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: audit
  Audit & Risk Logging (auth events, admin actions, risk decisions)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [17/20]: audit/01-auth_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[OK] Table audit/01-auth_event initialized

[INFO] 🔸 Table [18/20]: audit/02-admin_action
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[OK] Table audit/02-admin_action initialized

[INFO] 🔸 Table [19/20]: audit/03-risk_decision
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[OK] Table audit/03-risk_decision initialized

[INFO] 🔸 Table [20/20]: audit/04-consent_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[OK] Table audit/04-consent_event initialized


════════════════════════════════════════════════════════════════════════════
[OK] ✅ IAM Schema Initialization Complete!
[OK] All 20 tables initialized successfully

Schemas created:
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

Design highlights:
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

════════════════════════════════════════════════════════════════════════════

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=coordinator IDENTIFIER_PARENT=coordinator
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: identity/run.sh started for SERVICE=identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity
🔍 DEBUG_CHECKPOINT_A3: SERVICE_ROOT exists, discovering table folders
🔍 DEBUG_CHECKPOINT_A4: Found subfolder: auth
🔍 DEBUG_CHECKPOINT_A4b: Checking for nested schema layout in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth
🔍 DEBUG_CHECKPOINT_A4c: Found nested steps dir: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps (display: auth/login)
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: auth/login|/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
[INFO] 📚 Detected grouped table folders under identity/: auth/login

🔍 DEBUG_CHECKPOINT_A7: Current IDENTIFIER=coordinator
🔍 DEBUG_CHECKPOINT_A8_PROCEED: Processing tables on coordinator/main node
🔍 DEBUG_CHECKPOINT_A9: Processing table: auth/login at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
[INFO] 🔸 Table group: auth/login
🔍 DEBUG_CHECKPOINT_A10: About to run numbered steps for table: auth/login
🔍 DEBUG_CHECKPOINT_B1: run_all_numbered_steps_in_dir called for dir=/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps table=auth/login
🔍 DEBUG_CHECKPOINT_B2: Found 1 numbered steps: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B3: About to run step: 01-init-schema.sh
Ab substep 0 compelete start
[DEBUG] Tracking substep start: steps/01-install/steps/identity/auth/login/01-init-schema (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
Ab substep 0 compelete start
[INFO] 📦 01 init schema...
Ab substep 1 compelete start
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing auth.login_account table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating auth schema...
NOTICE:  schema "auth" already exists, skipping
CREATE SCHEMA
✅ Schema created
🔧 Creating account_status ENUM...
DO
✅ ENUM created
🔧 Creating auth.login_account table...
CREATE TABLE
✅ Table created (Citus-compatible with region_hint in all constraints)
🔧 Creating indexes...
CREATE INDEX
CREATE INDEX
✅ Indexes created
🔧 Creating Citus REFERENCE table for CDC compatibility...
 create_reference_table 
------------------------
 
(1 row)

✅ Table created as REFERENCE table (replicated to all nodes)
   CDC via Debezium will work correctly on coordinator
🎉 Schema initialization complete for fastorder_identity_sau_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

📊 Registering environment in monitoring database (obs schema)...
   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
   Resource IP: 142.93.238.16
⚠️  Could not connect to monitoring database, skipping registration
   You can manually register later using:
   /opt/fastorder/bash/scripts/env_app_setup/setup/04-postgresql/steps/register-authN-af-aaaa1-dev.sh

==========================================
✅ Schema initialization complete!
==========================================
Ab substep 1 compelete end
Ab substep 2 compelete start
Ab substep 2 compelete end

🔍 DEBUG_CHECKPOINT_B4: Completed step: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B5: All numbered steps completed for /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A11: Completed numbered steps for table: auth/login
compeleted here

🔍 DEBUG_CHECKPOINT_A12: All tables processed
End of 04-postgresql/steps/01-install/steps/identity/run.sh

✓ ✅ Coordinator setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up 1 worker(s) (Citus data nodes)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up worker: worker-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-02 08:13:50 UTC] USER=unknown EUID=33 PID=1718684 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 08:13:50 UTC] USER=unknown EUID=33 PID=1718691 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 08:13:50 UTC] USER=unknown EUID=33 PID=1718702 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-02 08:13:50 UTC] USER=unknown EUID=33 PID=1718718 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 08:13:50 UTC] USER=unknown EUID=33 PID=1718742 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-02 08:13:50 UTC] USER=unknown EUID=33 PID=1718763 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4 JOB_UUID=0c5008d4-812c-463b-922c-ff476c8d9257

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.214
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01...
[INFO]   db-identity-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.214

[INFO]   ✅ db-identity-sau-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.214    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 08:13:53 UTC] USER=www-data EUID=0 PID=1718943 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:13:53 UTC] USER=www-data EUID=0 PID=1718952 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-02 08:13:53 UTC] USER=www-data EUID=0 PID=1718962 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1718909
[2026-01-02 08:13:53 UTC] USER=www-data EUID=0 PID=1718971 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1718909/ra_root.crt
[2026-01-02 08:13:53 UTC] USER=www-data EUID=0 PID=1718980 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1718909/ra_root.key
[2026-01-02 08:13:54 UTC] USER=www-data EUID=0 PID=1718989 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1718909/ra_root.crt
[2026-01-02 08:13:54 UTC] USER=www-data EUID=0 PID=1718998 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1718909/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[2026-01-02 08:13:57 UTC] USER=www-data EUID=0 PID=1719049 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1718909/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 08:13:58 UTC] USER=www-data EUID=0 PID=1719058 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1718909/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 08:13:58 UTC] USER=www-data EUID=0 PID=1719067 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 08:13:58 UTC] USER=www-data EUID=0 PID=1719076 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1718909/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:13:58 UTC] USER=www-data EUID=0 PID=1719085 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:13:58 UTC] USER=www-data EUID=0 PID=1719095 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:13:58 UTC] USER=www-data EUID=0 PID=1719104 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 08:13:58 UTC] USER=www-data EUID=0 PID=1719115 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 08:13:58 UTC] USER=www-data EUID=0 PID=1719124 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 08:13:58 UTC] USER=www-data EUID=0 PID=1719133 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 08:13:58 UTC] USER=www-data EUID=0 PID=1719142 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 08:13:58 UTC] USER=www-data EUID=0 PID=1719151 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:13:58 UTC] USER=www-data EUID=0 PID=1719160 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01
Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719217 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719226 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719238 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719247 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719256 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719272 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719281 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719290 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719301 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719310 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719319 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719328 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719337 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719346 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719355 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719364 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719373 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719382 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719391 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719400 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:13:59 UTC] USER=www-data EUID=0 PID=1719409 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719437 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719446 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719455 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719464 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719473 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719483 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719492 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719501 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719510 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719519 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719528 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719538 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719548 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719557 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719566 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719575 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719584 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719593 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719602 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719611 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719620 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719629 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719638 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:14:00 UTC] USER=www-data EUID=0 PID=1719648 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719658 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719667 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719676 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719687 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719696 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719705 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719714 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719723 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719732 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719741 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719750 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719760 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719770 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719779 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719789 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719799 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719808 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719817 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719844 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719854 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719863 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:14:01 UTC] USER=www-data EUID=0 PID=1719872 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:14:02 UTC] USER=www-data EUID=0 PID=1719881 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:14:02 UTC] USER=www-data EUID=0 PID=1719891 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:14:02 UTC] USER=www-data EUID=0 PID=1719903 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:14:02 UTC] USER=www-data EUID=0 PID=1719912 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:14:02 UTC] USER=www-data EUID=0 PID=1719921 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:14:02 UTC] USER=www-data EUID=0 PID=1719930 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:14:02 UTC] USER=www-data EUID=0 PID=1719939 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:14:02 UTC] USER=www-data EUID=0 PID=1719948 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:14:02 UTC] USER=www-data EUID=0 PID=1719957 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:14:02 UTC] USER=www-data EUID=0 PID=1719966 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:14:02 UTC] USER=www-data EUID=0 PID=1719975 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:14:03 UTC] USER=www-data EUID=0 PID=1720018 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-01-02 08:14:03 UTC] USER=www-data EUID=0 PID=1720027 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 08:14:03 UTC] USER=www-data EUID=0 PID=1720036 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-01-02 08:14:03 UTC] USER=www-data EUID=0 PID=1720045 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 08:14:03 UTC] USER=www-data EUID=0 PID=1720054 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:14:03 UTC] USER=www-data EUID=0 PID=1720070 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:14:03 UTC] USER=www-data EUID=0 PID=1720079 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:14:03 UTC] USER=www-data EUID=0 PID=1720088 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:14:03 UTC] USER=www-data EUID=0 PID=1720097 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:14:03 UTC] USER=www-data EUID=0 PID=1720106 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:14:03 UTC] USER=www-data EUID=0 PID=1720115 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:14:03 UTC] USER=www-data EUID=0 PID=1720124 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:14:03 UTC] USER=www-data EUID=0 PID=1720133 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:14:03 UTC] USER=www-data EUID=0 PID=1720142 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720153 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720162 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720171 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720180 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720189 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720198 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720207 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720216 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720227 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720253 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720262 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720271 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720280 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720289 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720298 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720307 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720316 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720325 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720334 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720343 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720353 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720363 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720372 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720381 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720390 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:04 UTC] USER=www-data EUID=0 PID=1720399 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720408 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720417 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720426 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720435 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720444 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720453 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720463 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720473 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720485 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720513 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720543 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720574 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720605 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720625 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720634 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720644 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720653 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720664 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720674 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720684 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720693 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720704 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720715 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720725 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:14:05 UTC] USER=www-data EUID=0 PID=1720752 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:14:06 UTC] USER=www-data EUID=0 PID=1720761 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:14:06 UTC] USER=www-data EUID=0 PID=1720772 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:14:06 UTC] USER=www-data EUID=0 PID=1720781 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:14:06 UTC] USER=www-data EUID=0 PID=1720793 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:14:06 UTC] USER=www-data EUID=0 PID=1720803 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:14:06 UTC] USER=www-data EUID=0 PID=1720812 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:14:06 UTC] USER=www-data EUID=0 PID=1720821 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:14:06 UTC] USER=www-data EUID=0 PID=1720832 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:14:06 UTC] USER=www-data EUID=0 PID=1720842 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:14:06 UTC] USER=www-data EUID=0 PID=1720851 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:14:06 UTC] USER=www-data EUID=0 PID=1720860 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:14:06 UTC] USER=www-data EUID=0 PID=1720869 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:14:06 UTC] USER=www-data EUID=0 PID=1720878 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-postgresql environment: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com (10.100.1.214)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.214
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01
[INFO] Data dir:   /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[INFO] Port:       5432
[INFO] Hostname:   db-identity-sau-main-dev-postgresql-worker-01
[2026-01-02 08:14:08 UTC] USER=www-data EUID=0 PID=1720977 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:14:08 UTC] USER=www-data EUID=0 PID=1720998 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:14:08 UTC] USER=www-data EUID=0 PID=1721020 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 08:14:09 UTC] USER=www-data EUID=0 PID=1721082 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:14:09 UTC] USER=www-data EUID=0 PID=1721093 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-02 08:14:09 UTC] USER=www-data EUID=0 PID=1721103 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1721048
[2026-01-02 08:14:09 UTC] USER=www-data EUID=0 PID=1721112 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1721048/ra_root.crt
[2026-01-02 08:14:09 UTC] USER=www-data EUID=0 PID=1721121 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1721048/ra_root.key
[2026-01-02 08:14:09 UTC] USER=www-data EUID=0 PID=1721130 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1721048/ra_root.crt
[2026-01-02 08:14:09 UTC] USER=www-data EUID=0 PID=1721139 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1721048/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[2026-01-02 08:14:13 UTC] USER=www-data EUID=0 PID=1721194 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1721048/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 08:14:13 UTC] USER=www-data EUID=0 PID=1721203 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1721048/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 08:14:13 UTC] USER=www-data EUID=0 PID=1721213 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 08:14:13 UTC] USER=www-data EUID=0 PID=1721222 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1721048/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:14:13 UTC] USER=www-data EUID=0 PID=1721231 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:14:13 UTC] USER=www-data EUID=0 PID=1721240 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:14:13 UTC] USER=www-data EUID=0 PID=1721249 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 08:14:13 UTC] USER=www-data EUID=0 PID=1721260 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 08:14:13 UTC] USER=www-data EUID=0 PID=1721269 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 08:14:13 UTC] USER=www-data EUID=0 PID=1721278 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 08:14:13 UTC] USER=www-data EUID=0 PID=1721287 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 08:14:13 UTC] USER=www-data EUID=0 PID=1721296 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:14:13 UTC] USER=www-data EUID=0 PID=1721305 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01
Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-02 08:14:13 UTC] USER=www-data EUID=0 PID=1721335 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 08:14:13 UTC] USER=www-data EUID=0 PID=1721344 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 08:14:13 UTC] USER=www-data EUID=0 PID=1721353 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-02 08:14:14 UTC] USER=www-data EUID=0 PID=1721374 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-worker-01.service
[2026-01-02 08:14:14 UTC] USER=www-data EUID=0 PID=1721400 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-02 08:14:14 UTC] USER=www-data EUID=0 PID=1721433 ACTION=fsop ARGS=rm -rf /var/run/postgresql-identity-sau-main-dev-worker-01
[OK]   No conflicting Postgres left on port 5432
[OK]   Using postgres password from vault provider
[2026-01-02 08:14:16 UTC] USER=www-data EUID=0 PID=1721492 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.9su54G
[2026-01-02 08:14:16 UTC] USER=www-data EUID=0 PID=1721513 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.9su54G
[2026-01-02 08:14:16 UTC] USER=www-data EUID=0 PID=1721537 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 08:14:17 UTC] USER=www-data EUID=0 PID=1721559 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 08:14:17 UTC] USER=www-data EUID=0 PID=1721582 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/identity-sau-main-dev
[INFO] Initializing cluster in /var/lib/postgresql/17/identity-sau-main-dev/worker-01 (SCRAM; pwfile)
[WARN] Removing existing data directory: /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 08:14:17 UTC] USER=www-data EUID=0 PID=1721603 ACTION=fsop ARGS=rm -rf /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 08:14:17 UTC] USER=www-data EUID=0 PID=1721625 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 08:14:17 UTC] USER=www-data EUID=0 PID=1721646 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 08:14:17 UTC] USER=www-data EUID=0 PID=1721667 ACTION=fsop ARGS=chmod 700 /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 08:14:17 UTC] USER=www-data EUID=0 PID=1721688 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-02 08:14:17 UTC] USER=www-data EUID=0 PID=1721709 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-02 08:14:17 UTC] USER=www-data EUID=0 PID=1721732 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-02 08:14:17 UTC] USER=www-data EUID=0 PID=1721741 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /var/lib/postgresql/17/identity-sau-main-dev/worker-01 --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.9su54G
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/17/identity-sau-main-dev/worker-01 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /var/lib/postgresql/17/identity-sau-main-dev/worker-01 -l logfile start

[OK]   initdb complete
[2026-01-02 08:14:19 UTC] USER=www-data EUID=0 PID=1721787 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.9su54G
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-02 08:14:19 UTC] USER=www-data EUID=0 PID=1721837 ACTION=fsop ARGS=cp /tmp/tmp.HBP0ha1hvX /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 08:14:19 UTC] USER=www-data EUID=0 PID=1721858 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 08:14:19 UTC] USER=www-data EUID=0 PID=1721879 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01.service
[2026-01-02 08:14:19 UTC] USER=www-data EUID=0 PID=1721905 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.Ish85l /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01.service
[2026-01-02 08:14:19 UTC] USER=www-data EUID=0 PID=1721926 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01.service
[OK]   systemd unit written
[2026-01-02 08:14:20 UTC] USER=www-data EUID=0 PID=1721947 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 08:14:20 UTC] USER=www-data EUID=0 PID=1721968 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 08:14:20 UTC] USER=www-data EUID=0 PID=1721989 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-02 08:14:22 UTC] USER=www-data EUID=0 PID=1722116 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-worker-01.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-02 08:14:23 UTC] USER=www-data EUID=0 PID=1722156 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-worker-01.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Provisioning application database and Debezium role (if not exists)...
[INFO] Checking if database fastorder_identity_sau_main_dev_db exists...
[INFO] DB check result: exit_code=0, output='[2026-01-02 08:14:24 UTC] USER=www-data EUID=0 PID=1722319 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_identity_sau_main_dev_db''
[INFO] Creating database fastorder_identity_sau_main_dev_db...
[2026-01-02 08:14:24 UTC] USER=www-data EUID=0 PID=1722342 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_identity_sau_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[OK]   Database fastorder_identity_sau_main_dev_db created
[INFO] Checking if role debezium_user exists...
[INFO] Role check result: exit_code=0, output='[2026-01-02 08:14:24 UTC] USER=www-data EUID=0 PID=1722367 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[INFO] Creating role debezium_user...
[2026-01-02 08:14:25 UTC] USER=www-data EUID=0 PID=1722394 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'f5FtsvZu7cHjIhsEoWBhpvUE';
CREATE ROLE
[OK]   Role debezium_user created
[2026-01-02 08:14:25 UTC] USER=www-data EUID=0 PID=1722417 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_identity_sau_main_dev_db" TO debezium_user;
GRANT
[OK]   Application DB (fastorder_identity_sau_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[INFO] Applying connection and memory optimizations...
[INFO] Current settings: max_connections=100, work_mem=4MB
[INFO] Target settings (worker): max_connections=100, work_mem=8MB
[2026-01-02 08:14:25 UTC] USER=www-data EUID=0 PID=1722498 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 100;
ALTER SYSTEM
[2026-01-02 08:14:25 UTC] USER=www-data EUID=0 PID=1722521 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-02 08:14:26 UTC] USER=www-data EUID=0 PID=1722546 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   Settings applied to postgresql.auto.conf
[2026-01-02 08:14:26 UTC] USER=www-data EUID=0 PID=1722561 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01/standby.signal
[INFO] Service recently started (3s ago) - restarting to apply max_connections...
[INFO] Stopping service...
[2026-01-02 08:14:26 UTC] USER=www-data EUID=0 PID=1722583 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-worker-01.service
[INFO] Waiting for port 5432 to be released...
[OK]   Port 5432 released
[INFO] Starting service...
[2026-01-02 08:14:29 UTC] USER=www-data EUID=0 PID=1722631 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-worker-01.service
[2026-01-02 08:14:35 UTC] USER=www-data EUID=0 PID=1722794 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-worker-01.service
[OK]   ✅ Optimization complete: max_connections=100, work_mem=8MB
[OK]   Synchronous replication already configured (synchronous_commit: on)
[INFO] Setting postgres password via centralized script... for worker-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Initial setup: Using password from initdb
✓ PostgreSQL password already set during initdb
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/worker-01
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
[OK]   Password set and persisted
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.214
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01...
[INFO]   db-identity-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.214

[INFO]   ✅ db-identity-sau-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.214    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com


[OK]   PostgreSQL 'identity-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key \
        host=db-identity-sau-main-dev-postgresql-worker-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-worker-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.214
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-worker-01
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 2a8d7237-0c1b-4286-8ffc-cd46f4f7052e
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 📦 03 role...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 08:14:48 UTC] USER=www-data EUID=0 PID=1723200 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    debezium_user
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   debezium_user
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:15:17 UTC] USER=www-data EUID=0 PID=1727877 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-debezium_user
[2026-01-02 08:15:17 UTC] USER=www-data EUID=0 PID=1727907 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-01-02 08:15:17 UTC] USER=www-data EUID=0 PID=1727959 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-debezium_user/ra_root.key
[2026-01-02 08:15:17 UTC] USER=www-data EUID=0 PID=1727994 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-01-02 08:15:17 UTC] USER=www-data EUID=0 PID=1728015 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:15:17 UTC] USER=www-data EUID=0 PID=1728112 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:15:17 UTC] USER=www-data EUID=0 PID=1728192 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:15:18 UTC] USER=www-data EUID=0 PID=1728246 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 08:15:18 UTC] USER=www-data EUID=0 PID=1728315 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 08:15:18 UTC] USER=www-data EUID=0 PID=1728378 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:18 UTC] USER=www-data EUID=0 PID=1728411 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:15:18 UTC] USER=www-data EUID=0 PID=1728443 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 08:15:18 UTC] USER=www-data EUID=0 PID=1728499 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 08:15:18 UTC] USER=www-data EUID=0 PID=1728530 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 08:15:18 UTC] USER=www-data EUID=0 PID=1728559 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:18 UTC] USER=www-data EUID=0 PID=1728579 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:15:18 UTC] USER=www-data EUID=0 PID=1728610 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 08:15:18 UTC] USER=www-data EUID=0 PID=1728645 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 08:15:18 UTC] USER=www-data EUID=0 PID=1728676 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 08:15:18 UTC] USER=www-data EUID=0 PID=1728712 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:19 UTC] USER=www-data EUID=0 PID=1728740 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:19 UTC] USER=www-data EUID=0 PID=1728843 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:19 UTC] USER=www-data EUID=0 PID=1728879 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:19 UTC] USER=www-data EUID=0 PID=1728931 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:19 UTC] USER=www-data EUID=0 PID=1728964 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:19 UTC] USER=www-data EUID=0 PID=1728993 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:19 UTC] USER=www-data EUID=0 PID=1729014 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 08:15:19 UTC] USER=www-data EUID=0 PID=1729099 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 08:15:19 UTC] USER=www-data EUID=0 PID=1729149 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:19 UTC] USER=www-data EUID=0 PID=1729197 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:15:19 UTC] USER=www-data EUID=0 PID=1729318 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 08:15:20 UTC] USER=www-data EUID=0 PID=1729423 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:20 UTC] USER=www-data EUID=0 PID=1729493 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:20 UTC] USER=www-data EUID=0 PID=1729535 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:20 UTC] USER=www-data EUID=0 PID=1729576 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:20 UTC] USER=www-data EUID=0 PID=1729636 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:20 UTC] USER=www-data EUID=0 PID=1729677 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:20 UTC] USER=www-data EUID=0 PID=1729731 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 08:15:20 UTC] USER=www-data EUID=0 PID=1729776 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 08:15:20 UTC] USER=www-data EUID=0 PID=1729820 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:20 UTC] USER=www-data EUID=0 PID=1729858 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:15:20 UTC] USER=www-data EUID=0 PID=1729920 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 08:15:20 UTC] USER=www-data EUID=0 PID=1729967 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 08:15:20 UTC] USER=www-data EUID=0 PID=1730028 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:20 UTC] USER=www-data EUID=0 PID=1730113 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:21 UTC] USER=www-data EUID=0 PID=1730187 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:21 UTC] USER=www-data EUID=0 PID=1730231 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:21 UTC] USER=www-data EUID=0 PID=1730281 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:21 UTC] USER=www-data EUID=0 PID=1730332 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:21 UTC] USER=www-data EUID=0 PID=1730382 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 08:15:21 UTC] USER=www-data EUID=0 PID=1730433 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 08:15:21 UTC] USER=www-data EUID=0 PID=1730502 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:21 UTC] USER=www-data EUID=0 PID=1730575 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:15:21 UTC] USER=www-data EUID=0 PID=1730640 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 08:15:21 UTC] USER=www-data EUID=0 PID=1730702 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 08:15:21 UTC] USER=www-data EUID=0 PID=1730752 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:21 UTC] USER=www-data EUID=0 PID=1730856 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:21 UTC] USER=www-data EUID=0 PID=1730918 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:21 UTC] USER=www-data EUID=0 PID=1731072 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:22 UTC] USER=www-data EUID=0 PID=1731211 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:22 UTC] USER=www-data EUID=0 PID=1731279 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 08:15:22 UTC] USER=www-data EUID=0 PID=1731349 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 08:15:22 UTC] USER=www-data EUID=0 PID=1731420 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:22 UTC] USER=www-data EUID=0 PID=1731503 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:15:22 UTC] USER=www-data EUID=0 PID=1731564 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 08:15:22 UTC] USER=www-data EUID=0 PID=1731589 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 08:15:22 UTC] USER=www-data EUID=0 PID=1731599 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:15:22 UTC] USER=www-data EUID=0 PID=1731609 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:15:22 UTC] USER=www-data EUID=0 PID=1731618 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:15:22 UTC] USER=www-data EUID=0 PID=1731627 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:15:22 UTC] USER=www-data EUID=0 PID=1731636 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:15:22 UTC] USER=www-data EUID=0 PID=1731645 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:15:22 UTC] USER=www-data EUID=0 PID=1731654 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:15:22 UTC] USER=www-data EUID=0 PID=1731663 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:15:22 UTC] USER=www-data EUID=0 PID=1731672 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:15:22 UTC] USER=www-data EUID=0 PID=1731681 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: debezium_user
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U debezium_user -d postgres

🔐 Generating replicator client certificate for worker-01...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:15:23 UTC] USER=www-data EUID=0 PID=1731723 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-02 08:15:23 UTC] USER=www-data EUID=0 PID=1731732 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 08:15:23 UTC] USER=www-data EUID=0 PID=1731741 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-02 08:15:23 UTC] USER=www-data EUID=0 PID=1731750 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 08:15:23 UTC] USER=www-data EUID=0 PID=1731759 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:15:23 UTC] USER=www-data EUID=0 PID=1731775 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:15:23 UTC] USER=www-data EUID=0 PID=1731784 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:15:23 UTC] USER=www-data EUID=0 PID=1731794 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731803 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731812 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731821 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731832 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731841 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731850 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731859 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731868 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731877 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731886 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731896 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731907 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731916 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731925 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731934 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731960 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731969 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731978 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731987 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1731996 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1732005 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1732014 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1732023 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1732032 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1732041 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:15:24 UTC] USER=www-data EUID=0 PID=1732050 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732060 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732070 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732079 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732088 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732097 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732106 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732115 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732124 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732133 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732142 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732151 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732160 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732170 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732180 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732189 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732198 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732207 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732216 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732225 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732236 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732245 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732254 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732263 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732272 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732282 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732292 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:25 UTC] USER=www-data EUID=0 PID=1732301 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732310 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732319 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732328 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732338 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732350 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732359 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732368 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732377 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732386 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732396 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732406 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732415 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732424 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732433 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732442 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732451 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732461 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732470 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:15:26 UTC] USER=www-data EUID=0 PID=1732479 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

✅ Replicator certificate generated for worker-01
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: fastorder_admin_gd
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: worker-01/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    fastorder_admin_gd
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   fastorder_admin_gd
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:15:42 UTC] USER=www-data EUID=0 PID=1733023 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-01-02 08:15:42 UTC] USER=www-data EUID=0 PID=1733034 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
[2026-01-02 08:15:42 UTC] USER=www-data EUID=0 PID=1733043 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-01-02 08:15:42 UTC] USER=www-data EUID=0 PID=1733052 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:15:42 UTC] USER=www-data EUID=0 PID=1733067 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:15:42 UTC] USER=www-data EUID=0 PID=1733076 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:15:42 UTC] USER=www-data EUID=0 PID=1733085 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 08:15:42 UTC] USER=www-data EUID=0 PID=1733094 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733103 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733112 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733121 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733130 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733139 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733149 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733158 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733167 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733176 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733185 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733196 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733205 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733214 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733223 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733249 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733258 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733267 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733276 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733285 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733295 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733305 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733314 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733323 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733332 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:15:43 UTC] USER=www-data EUID=0 PID=1733341 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733351 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733361 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733370 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733379 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733388 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733397 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733406 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733415 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733424 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733433 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733442 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733453 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733463 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733473 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733482 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733491 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733500 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733509 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733518 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733527 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733536 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733545 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733554 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733563 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733573 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733583 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733592 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733601 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733610 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:44 UTC] USER=www-data EUID=0 PID=1733619 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:15:45 UTC] USER=www-data EUID=0 PID=1733628 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 08:15:45 UTC] USER=www-data EUID=0 PID=1733637 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 08:15:45 UTC] USER=www-data EUID=0 PID=1733646 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:15:45 UTC] USER=www-data EUID=0 PID=1733655 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:15:45 UTC] USER=www-data EUID=0 PID=1733664 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:15:45 UTC] USER=www-data EUID=0 PID=1733673 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 08:15:45 UTC] USER=www-data EUID=0 PID=1733683 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:15:45 UTC] USER=www-data EUID=0 PID=1733693 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:15:45 UTC] USER=www-data EUID=0 PID=1733702 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:15:45 UTC] USER=www-data EUID=0 PID=1733711 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:15:45 UTC] USER=www-data EUID=0 PID=1733720 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:15:45 UTC] USER=www-data EUID=0 PID=1733731 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:15:45 UTC] USER=www-data EUID=0 PID=1733740 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:15:45 UTC] USER=www-data EUID=0 PID=1733749 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:15:45 UTC] USER=www-data EUID=0 PID=1733758 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:15:45 UTC] USER=www-data EUID=0 PID=1733767 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: fastorder_admin_gd
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-identity-sau-main-dev-worker-01:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_identity_sau_main_dev_db already exists, skipping creation
[2026-01-02 08:15:45 UTC] USER=www-data EUID=0 PID=1733828 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-02 08:15:46 UTC] USER=www-data EUID=0 PID=1733865 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

=== Pre-flight Checks ===
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
✓ AWS Secrets Manager accessible

=== Retrieving Credentials from AWS ===
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
/opt/fastorder/bash/infra_core/cache.sh: line 145: /var/cache/secrets/fastorder_db_identity_sau_main_dev_postgresql_worker-01_fastorder_admin_gd.cache.tmp.1733879: Permission denied
✅ Retrieved from secrets manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials loaded for worker-01/fastorder_admin_gd: fastorder_admin_gd@db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_identity_sau_main_dev_db
✓ Credentials retrieved: fastorder_admin_gd@db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_identity_sau_main_dev_db
╔════════════════════════════════════════════╗
║  PostgreSQL Test Suite (AWS Secrets MGR)  ║
╚════════════════════════════════════════════╝

=== PostgreSQL Authentication Test ===
✗ PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-identity-sau-main-dev-postgresql-worker-01.fastorder.com" (10.100.1.214), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 08:15:54 UTC] USER=www-data EUID=0 PID=1734170 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_identity_sau_main_dev_db
  SCHEMA      : auth
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
  DNS → 10.100.1.214
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User debezium_user does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: debezium_user
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: worker-01/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-01-02 08:16:08 UTC] USER=www-data EUID=0 PID=1734628 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
[2026-01-02 08:16:08 UTC] USER=www-data EUID=0 PID=1734654 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d fastorder_identity_sau_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
🔐 Creating replicator role for worker-01...
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
  DNS → 10.100.1.214
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Creating role: replicator with password
SET
CREATE ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/identity/sau/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🎉 Done.
✅ Replicator role created for worker-01

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=worker-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║                    IAM Database Schema Initialization                       ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] 🟢 Starting IAM schema provisioning...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

[INFO] 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: core
  Core Identity Directory (tenants, realms, identities, devices, MFA)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [1/20]: core/01-tenant
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "pgcrypto" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "citext" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
NOTICE:  schema "utils" already exists, skipping
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
NOTICE:  schema "core" already exists, skipping
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
NOTICE:  relation "tenant" already exists, skipping
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[OK] Table core/01-tenant initialized

[INFO] 🔸 Table [2/20]: core/02-realm
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  relation "realm" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_realm_keycloak_id" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_realm_tenant" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[OK] Table core/02-realm initialized

[INFO] 🔸 Table [3/20]: core/03-identity
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  relation "identity" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_unique_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_unique_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_type" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[OK] Table core/03-identity initialized

[INFO] 🔸 Table [4/20]: core/04-device
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  relation "device" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_device_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_fingerprint" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_trusted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_last_seen" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[OK] Table core/04-device initialized

[INFO] 🔸 Table [5/20]: core/05-identity_account
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  relation "identity_account" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_account_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_lockout" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_last_login" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[OK] Table core/05-identity_account initialized

[INFO] 🔸 Table [6/20]: core/06-identity_mfa
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  relation "identity_mfa" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_mfa_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_active" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[OK] Table core/06-identity_mfa initialized

[INFO] 🔸 Table [7/20]: core/07-external_idp_link
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  relation "external_idp_link" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_external_idp_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_provider" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_email" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[OK] Table core/07-external_idp_link initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: policy
  RBAC/ABAC Authorization (clients, roles, permissions, policies)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [8/20]: policy/01-client
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
NOTICE:  schema "policy" already exists, skipping
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  relation "client" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_client_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_key" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_status" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[OK] Table policy/01-client initialized

[INFO] 🔸 Table [9/20]: policy/02-resource
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  relation "resource" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_resource_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_external" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_owner" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[OK] Table policy/02-resource initialized

[INFO] 🔸 Table [10/20]: policy/03-scope
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  relation "scope" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_scope_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_scope_name" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[OK] Table policy/03-scope initialized

[INFO] 🔸 Table [11/20]: policy/04-permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  relation "permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_permission_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_resource" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[OK] Table policy/04-permission initialized

[INFO] 🔸 Table [12/20]: policy/05-role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  relation "role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_keycloak" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[OK] Table policy/05-role initialized

[INFO] 🔸 Table [13/20]: policy/06-role_permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  relation "role_permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_permission_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_permission_perm" already exists, skipping
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[OK] Table policy/06-role_permission initialized

[INFO] 🔸 Table [14/20]: policy/07-identity_role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  relation "identity_role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_role_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_active" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[OK] Table policy/07-identity_role initialized

[INFO] 🔸 Table [15/20]: policy/08-policy_rule
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  relation "policy_rule" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_policy_rule_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_enabled" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_priority" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[OK] Table policy/08-policy_rule initialized

[INFO] 🔸 Table [16/20]: policy/09-api_key
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  relation "api_key" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_api_key_prefix" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[OK] Table policy/09-api_key initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: audit
  Audit & Risk Logging (auth events, admin actions, risk decisions)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [17/20]: audit/01-auth_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
NOTICE:  schema "audit" already exists, skipping
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
NOTICE:  relation "auth_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_auth_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_result" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_ip" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_session" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_trace" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_risk" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[OK] Table audit/01-auth_event initialized

[INFO] 🔸 Table [18/20]: audit/02-admin_action
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
NOTICE:  relation "admin_action" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_admin_action_actor" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_target" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_trace" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[OK] Table audit/02-admin_action initialized

[INFO] 🔸 Table [19/20]: audit/03-risk_decision
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
NOTICE:  relation "risk_decision" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_risk_decision_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_level" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_decision" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_auth" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[OK] Table audit/03-risk_decision initialized

[INFO] 🔸 Table [20/20]: audit/04-consent_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
NOTICE:  relation "consent_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_consent_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_version" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_granted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  relation "audit.auth_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  relation "audit.auth_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  relation "audit.admin_action_2026_03" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  relation "audit.admin_action_2026_04" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  relation "audit.risk_decision_2026_03" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  relation "audit.risk_decision_2026_04" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  relation "audit.consent_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  relation "audit.consent_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[OK] Table audit/04-consent_event initialized


════════════════════════════════════════════════════════════════════════════
[OK] ✅ IAM Schema Initialization Complete!
[OK] All 20 tables initialized successfully

Schemas created:
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

Design highlights:
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

════════════════════════════════════════════════════════════════════════════

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=worker-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: identity/run.sh started for SERVICE=identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity
🔍 DEBUG_CHECKPOINT_A3: SERVICE_ROOT exists, discovering table folders
🔍 DEBUG_CHECKPOINT_A4: Found subfolder: auth
🔍 DEBUG_CHECKPOINT_A4b: Checking for nested schema layout in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth
🔍 DEBUG_CHECKPOINT_A4c: Found nested steps dir: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps (display: auth/login)
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: auth/login|/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
[INFO] 📚 Detected grouped table folders under identity/: auth/login

🔍 DEBUG_CHECKPOINT_A7: Current IDENTIFIER=coordinator
🔍 DEBUG_CHECKPOINT_A8_PROCEED: Processing tables on coordinator/main node
🔍 DEBUG_CHECKPOINT_A9: Processing table: auth/login at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
[INFO] 🔸 Table group: auth/login
🔍 DEBUG_CHECKPOINT_A10: About to run numbered steps for table: auth/login
🔍 DEBUG_CHECKPOINT_B1: run_all_numbered_steps_in_dir called for dir=/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps table=auth/login
🔍 DEBUG_CHECKPOINT_B2: Found 1 numbered steps: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B3: About to run step: 01-init-schema.sh
Ab substep 0 compelete start
[DEBUG] Tracking substep start: steps/01-install/steps/identity/auth/login/01-init-schema (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
Ab substep 0 compelete start
[INFO] 📦 01 init schema...
Ab substep 1 compelete start
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing auth.login_account table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating auth schema...
NOTICE:  schema "auth" already exists, skipping
CREATE SCHEMA
✅ Schema created
🔧 Creating account_status ENUM...
DO
✅ ENUM created
🔧 Creating auth.login_account table...
NOTICE:  relation "login_account" already exists, skipping
CREATE TABLE
✅ Table created (Citus-compatible with region_hint in all constraints)
🔧 Creating indexes...
NOTICE:  relation "idx_login_account_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_login_account_username" already exists, skipping
CREATE INDEX
✅ Indexes created
ℹ️  Table already registered with Citus
🎉 Schema initialization complete for fastorder_identity_sau_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

📊 Registering environment in monitoring database (obs schema)...
   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
   Resource IP: 142.93.238.16
⚠️  Could not connect to monitoring database, skipping registration
   You can manually register later using:
   /opt/fastorder/bash/scripts/env_app_setup/setup/04-postgresql/steps/register-authN-af-aaaa1-dev.sh

==========================================
✅ Schema initialization complete!
==========================================
Ab substep 1 compelete end
Ab substep 2 compelete start
Ab substep 2 compelete end

🔍 DEBUG_CHECKPOINT_B4: Completed step: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B5: All numbered steps completed for /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A11: Completed numbered steps for table: auth/login
compeleted here

🔍 DEBUG_CHECKPOINT_A12: All tables processed
End of 04-postgresql/steps/01-install/steps/identity/run.sh

✓ ✅ Worker worker-01 setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up standby replicas (1 per worker)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up standby: worker-01-standby-01 (replica of worker-01)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-02 08:17:47 UTC] USER=unknown EUID=33 PID=1738227 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 08:17:47 UTC] USER=unknown EUID=33 PID=1738234 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 08:17:47 UTC] USER=unknown EUID=33 PID=1738241 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-02 08:17:47 UTC] USER=unknown EUID=33 PID=1738248 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 08:17:47 UTC] USER=unknown EUID=33 PID=1738255 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-02 08:17:47 UTC] USER=unknown EUID=33 PID=1738262 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4 JOB_UUID=0c5008d4-812c-463b-922c-ff476c8d9257

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] PostgreSQL IP: 10.100.1.211
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01-standby-01...
[INFO]   db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.211

[INFO]   ✅ db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.211    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  identity-sau-main-dev.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 08:17:50 UTC] USER=www-data EUID=0 PID=1738396 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:51 UTC] USER=www-data EUID=0 PID=1738405 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-02 08:17:51 UTC] USER=www-data EUID=0 PID=1738415 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1738357
[2026-01-02 08:17:51 UTC] USER=www-data EUID=0 PID=1738424 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1738357/ra_root.crt
[2026-01-02 08:17:51 UTC] USER=www-data EUID=0 PID=1738433 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1738357/ra_root.key
[2026-01-02 08:17:51 UTC] USER=www-data EUID=0 PID=1738442 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1738357/ra_root.crt
[2026-01-02 08:17:51 UTC] USER=www-data EUID=0 PID=1738453 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1738357/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
[2026-01-02 08:17:53 UTC] USER=www-data EUID=0 PID=1738595 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1738357/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 08:17:53 UTC] USER=www-data EUID=0 PID=1738604 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1738357/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 08:17:53 UTC] USER=www-data EUID=0 PID=1738613 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 08:17:53 UTC] USER=www-data EUID=0 PID=1738622 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1738357/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:17:53 UTC] USER=www-data EUID=0 PID=1738631 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:17:53 UTC] USER=www-data EUID=0 PID=1738640 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:17:53 UTC] USER=www-data EUID=0 PID=1738649 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 08:17:53 UTC] USER=www-data EUID=0 PID=1738660 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 08:17:53 UTC] USER=www-data EUID=0 PID=1738669 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 08:17:54 UTC] USER=www-data EUID=0 PID=1738679 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 08:17:54 UTC] USER=www-data EUID=0 PID=1738688 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 08:17:54 UTC] USER=www-data EUID=0 PID=1738697 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:54 UTC] USER=www-data EUID=0 PID=1738706 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:identity-sau-main-dev.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01-standby-01
Primary CN:  identity-sau-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=identity-sau-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:17:54 UTC] USER=www-data EUID=0 PID=1738764 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-01-02 08:17:54 UTC] USER=www-data EUID=0 PID=1738773 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 08:17:54 UTC] USER=www-data EUID=0 PID=1738782 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
[2026-01-02 08:17:54 UTC] USER=www-data EUID=0 PID=1738791 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 08:17:54 UTC] USER=www-data EUID=0 PID=1738800 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738816 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738825 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738834 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738843 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738852 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738861 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738870 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738879 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738888 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738897 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738906 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738915 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738924 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738933 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738942 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738951 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738977 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738986 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1738995 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1739004 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1739013 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1739022 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:17:55 UTC] USER=www-data EUID=0 PID=1739031 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739040 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739049 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739058 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739067 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739077 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739089 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739098 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739107 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739116 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739125 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739134 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739143 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739152 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739161 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739170 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739179 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739191 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739202 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739211 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739238 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739247 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739256 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739265 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739274 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739283 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:17:56 UTC] USER=www-data EUID=0 PID=1739292 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739304 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739315 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739324 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739333 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739342 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739351 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739360 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739369 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739378 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739387 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739396 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739405 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739415 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739425 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739434 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739443 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739452 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739461 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739470 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739479 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739488 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:17:57 UTC] USER=www-data EUID=0 PID=1739497 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:17:58 UTC] USER=www-data EUID=0 PID=1739538 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-01-02 08:17:58 UTC] USER=www-data EUID=0 PID=1739547 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 08:17:58 UTC] USER=www-data EUID=0 PID=1739556 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
[2026-01-02 08:17:58 UTC] USER=www-data EUID=0 PID=1739565 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 08:17:58 UTC] USER=www-data EUID=0 PID=1739574 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739594 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739603 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739612 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739621 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739630 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739639 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739648 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739657 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739666 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739675 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739685 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739695 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739704 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739713 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739722 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739732 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739741 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739750 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739776 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739785 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739794 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739803 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739812 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739821 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739830 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739839 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:17:59 UTC] USER=www-data EUID=0 PID=1739849 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1739858 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1739867 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1739877 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1739887 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1739896 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1739905 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1739914 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1739923 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1739932 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1739941 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1739950 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1739959 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1739968 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1739977 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1739988 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1740000 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1740009 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1740018 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1740027 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1740036 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1740045 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1740054 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1740063 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1740072 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1740081 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:18:00 UTC] USER=www-data EUID=0 PID=1740090 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740100 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740113 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740122 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740131 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740140 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740151 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740160 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740169 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740178 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740187 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740198 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740207 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740218 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740237 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740246 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740255 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740264 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740273 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740282 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:18:01 UTC] USER=www-data EUID=0 PID=1740291 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:18:02 UTC] USER=www-data EUID=0 PID=1740362 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-02 08:18:02 UTC] USER=www-data EUID=0 PID=1740371 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 08:18:02 UTC] USER=www-data EUID=0 PID=1740380 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-02 08:18:02 UTC] USER=www-data EUID=0 PID=1740389 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 08:18:02 UTC] USER=www-data EUID=0 PID=1740398 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:18:02 UTC] USER=www-data EUID=0 PID=1740412 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:18:02 UTC] USER=www-data EUID=0 PID=1740421 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740430 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740439 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740448 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740457 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740466 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740475 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740484 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740493 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740502 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740511 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740520 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740529 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740540 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740549 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740558 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740567 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740593 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740602 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740611 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740620 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740629 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740638 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740649 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740658 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740667 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740676 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:18:03 UTC] USER=www-data EUID=0 PID=1740685 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740695 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740705 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740714 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740723 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740732 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740741 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740750 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740759 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740768 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740778 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740787 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740798 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740808 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740818 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740827 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740836 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740845 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740854 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740863 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740872 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740881 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740890 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740900 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740909 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740919 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740929 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:04 UTC] USER=www-data EUID=0 PID=1740938 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1740947 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1740956 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1740965 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1740974 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1740983 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1740992 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1741001 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1741010 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1741019 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1741029 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1741039 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1741050 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1741059 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1741068 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1741077 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1741086 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1741095 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1741104 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:18:05 UTC] USER=www-data EUID=0 PID=1741113 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-standby-01-postgresql environment: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com (10.100.1.211)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.211
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] Data dir:   /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[INFO] Port:       5432
[INFO] Hostname:   db-identity-sau-main-dev-postgresql-worker-01-standby-01
[2026-01-02 08:18:07 UTC] USER=www-data EUID=0 PID=1741312 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:07 UTC] USER=www-data EUID=0 PID=1741334 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:07 UTC] USER=www-data EUID=0 PID=1741357 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:08 UTC] USER=www-data EUID=0 PID=1741378 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  identity-sau-main-dev.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 08:18:08 UTC] USER=www-data EUID=0 PID=1741419 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:08 UTC] USER=www-data EUID=0 PID=1741428 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-02 08:18:08 UTC] USER=www-data EUID=0 PID=1741438 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1741385
[2026-01-02 08:18:08 UTC] USER=www-data EUID=0 PID=1741447 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1741385/ra_root.crt
[2026-01-02 08:18:08 UTC] USER=www-data EUID=0 PID=1741456 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1741385/ra_root.key
[2026-01-02 08:18:08 UTC] USER=www-data EUID=0 PID=1741465 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1741385/ra_root.crt
[2026-01-02 08:18:08 UTC] USER=www-data EUID=0 PID=1741474 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1741385/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
[2026-01-02 08:18:11 UTC] USER=www-data EUID=0 PID=1741539 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1741385/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 08:18:12 UTC] USER=www-data EUID=0 PID=1741549 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1741385/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 08:18:12 UTC] USER=www-data EUID=0 PID=1741558 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 08:18:12 UTC] USER=www-data EUID=0 PID=1741567 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1741385/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:18:12 UTC] USER=www-data EUID=0 PID=1741576 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:18:12 UTC] USER=www-data EUID=0 PID=1741585 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:18:12 UTC] USER=www-data EUID=0 PID=1741600 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 08:18:12 UTC] USER=www-data EUID=0 PID=1741612 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 08:18:12 UTC] USER=www-data EUID=0 PID=1741621 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 08:18:12 UTC] USER=www-data EUID=0 PID=1741630 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 08:18:12 UTC] USER=www-data EUID=0 PID=1741639 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 08:18:12 UTC] USER=www-data EUID=0 PID=1741648 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:12 UTC] USER=www-data EUID=0 PID=1741659 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:identity-sau-main-dev.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01-standby-01
Primary CN:  identity-sau-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=identity-sau-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-02 08:18:12 UTC] USER=www-data EUID=0 PID=1741693 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 08:18:12 UTC] USER=www-data EUID=0 PID=1741702 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 08:18:12 UTC] USER=www-data EUID=0 PID=1741711 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-02 08:18:12 UTC] USER=www-data EUID=0 PID=1741732 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-worker-01-standby-01.service
[2026-01-02 08:18:13 UTC] USER=www-data EUID=0 PID=1741758 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[2026-01-02 08:18:13 UTC] USER=www-data EUID=0 PID=1741789 ACTION=fsop ARGS=rm -rf /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[OK]   No conflicting Postgres left on port 5432
[OK]   Generated new postgres password for initdb
[2026-01-02 08:18:38 UTC] USER=www-data EUID=0 PID=1742075 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.UTx22n
[2026-01-02 08:18:38 UTC] USER=www-data EUID=0 PID=1742096 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.UTx22n
[2026-01-02 08:18:38 UTC] USER=www-data EUID=0 PID=1742118 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 08:18:38 UTC] USER=www-data EUID=0 PID=1742140 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 08:18:39 UTC] USER=www-data EUID=0 PID=1742162 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/identity-sau-main-dev
[INFO] This is a standby. Using pg_basebackup from primary (worker-01)...
[INFO] Setting up replicator role and slot on primary (worker-01)...
ℹ️  Scanning primary for stuck queries from previous failed attempts...
ℹ️  Scanning for stuck queries (timeout: 30s)...
ℹ️  No stuck queries found
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SLOT        : worker_01_standby_01
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
  DNS → 10.100.1.214
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Role replicator already exists, updating password and ensuring REPLICATION privilege
SET
ALTER ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/identity/sau/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🔧 Ensuring replication slot: worker_01_standby_01…
🆕 Creating replication slot worker_01_standby_01
SET
 pg_create_physical_replication_slot 
-------------------------------------
 (worker_01_standby_01,)
(1 row)

✅ Replication slot worker_01_standby_01 created.
🎉 Done.
[OK]   Replicator role and slot created on primary
[INFO] Creating replicator client certificates for connecting to primary (worker-01)...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:18:43 UTC] USER=www-data EUID=0 PID=1742333 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-02 08:18:43 UTC] USER=www-data EUID=0 PID=1742342 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 08:18:43 UTC] USER=www-data EUID=0 PID=1742351 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-02 08:18:43 UTC] USER=www-data EUID=0 PID=1742360 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 08:18:43 UTC] USER=www-data EUID=0 PID=1742369 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742386 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742395 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742404 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742413 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742424 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742434 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742445 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742454 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742463 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742472 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742481 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742490 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742499 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742508 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742517 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742526 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742535 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742544 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:44 UTC] USER=www-data EUID=0 PID=1742572 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742581 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742590 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742599 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742608 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742617 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742626 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742635 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742644 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742653 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742662 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742672 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742684 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742693 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742702 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742711 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742720 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742729 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742738 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742747 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742756 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742765 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742774 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742784 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742794 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742803 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742812 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742821 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742830 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:45 UTC] USER=www-data EUID=0 PID=1742839 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1742848 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1742857 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1742866 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1742875 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1742884 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1742894 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1742904 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1742913 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1742922 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1742931 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1742940 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1742949 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1742959 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1742968 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1742977 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1742986 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1742995 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1743005 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1743015 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1743024 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1743033 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1743042 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1743051 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1743060 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1743069 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1743078 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:18:46 UTC] USER=www-data EUID=0 PID=1743087 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

[OK]   Replicator certificate created for worker-01 in /home/postgres/
[INFO] Using replicator certificates from primary worker-01...
[2026-01-02 08:18:47 UTC] USER=www-data EUID=0 PID=1743117 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:18:47 UTC] USER=www-data EUID=0 PID=1743138 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:18:47 UTC] USER=www-data EUID=0 PID=1743159 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[OK]   Replicator certificates verified at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[OK]   root.crt verified at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[INFO] Updating primary pg_hba.conf to allow replication...
[INFO]   Standby IP: 10.100.1.211/32 (standby's source IP)
[INFO]   Primary application IP: 10.100.1.214/32 (for local pg_basebackup)
[INFO]   Primary DNS IP: 10.100.1.214/32 (DNS resolution of db-identity-sau-main-dev-postgresql-worker-01.fastorder.com)
[2026-01-02 08:18:47 UTC] USER=www-data EUID=0 PID=1743191 ACTION=passthru ARGS=grep -qxF # BEGIN standby-replication (managed) /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 08:18:47 UTC] USER=www-data EUID=0 PID=1743235 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.211/32  scram-sha-256 
      $0==begin {inside=1}
      inside && $0==rule {found=1}
      $0==end {inside=0}
      END {exit found?0:1}
     /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 08:18:47 UTC] USER=www-data EUID=0 PID=1743259 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl  replication  replicator  10.100.1.211/32  scram-sha-256 /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 08:18:47 UTC] USER=www-data EUID=0 PID=1743281 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.214/32  scram-sha-256 
        $0==begin {inside=1}
        inside && $0==rule {found=1}
        $0==end {inside=0}
        END {exit found?0:1}
       /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 08:18:48 UTC] USER=www-data EUID=0 PID=1743305 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl  replication  replicator  10.100.1.214/32  scram-sha-256 /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[INFO] Reloading primary PostgreSQL service...
[2026-01-02 08:18:48 UTC] USER=www-data EUID=0 PID=1743326 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
[OK]   Primary pg_hba.conf updated and service reloaded
[WARN] Removing existing data directory: /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:18:48 UTC] USER=www-data EUID=0 PID=1743349 ACTION=fsop ARGS=rm -rf /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[INFO] Primary host: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO] Using replicator cert: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[INFO] Using replicator key: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key (PKCS#8 format)
[INFO] Using CA cert: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[INFO] Verifying postgres user can access certificates...
[ERR]  postgres user CANNOT read /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[INFO] File permissions:
lrwxrwxrwx 1 postgres ssl-cert 72 Jan  2 08:18 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt -> /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[INFO] Parent directory permissions:
drwx------ 2 postgres postgres 4096 Jan  2 08:18 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
drwx------ 6 postgres postgres 4096 Jan  2 07:10 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[WARN] Attempting to fix permissions (/usr/local/bin/fastorder-provisioning-wrapper.sh required)...
[INFO] Fixing /home/postgres/ directory...
[2026-01-02 08:18:48 UTC] USER=www-data EUID=0 PID=1743417 ACTION=fsop ARGS=chmod 755 /home/postgres/
[INFO] Fixing /home/postgres/ssl/.postgresql/...
[2026-01-02 08:18:48 UTC] USER=www-data EUID=0 PID=1743438 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/
[INFO] Fixing parent directory: /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:18:48 UTC] USER=www-data EUID=0 PID=1743463 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[INFO] Fixing certificate directory: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:18:49 UTC] USER=www-data EUID=0 PID=1743484 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[INFO] Fixing CA certificate: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:18:49 UTC] USER=www-data EUID=0 PID=1743505 ACTION=fsop ARGS=chmod 644 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[OK]   Permissions fixed
[OK]   postgres user can now read /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt after permission fix
[2026-01-02 08:18:49 UTC] USER=www-data EUID=0 PID=1743526 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[2026-01-02 08:18:49 UTC] USER=www-data EUID=0 PID=1743549 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[2026-01-02 08:18:49 UTC] USER=www-data EUID=0 PID=1743572 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[INFO] Checking primary database size before pg_basebackup...
[INFO] Total primary database size: 29 MB
[INFO] Estimated transfer time: ~0 minutes (at 10MB/s with compression)
[INFO] Retrieving replicator password from AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/replicator
[OK]   Replicator password retrieved successfully
[INFO] Starting pg_basebackup...
[2026-01-02 08:18:52 UTC] USER=www-data EUID=0 PID=1743683 ACTION=passthru ARGS=sudo -u postgres env PGPASSWORD=qrzga0rZrBWHXjHNfE1t9bdwqo6QF84R PGSSLMODE=verify-full PGSSLCERT=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt PGSSLKEY=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key PGSSLROOTCERT=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /usr/lib/postgresql/17/bin/pg_basebackup -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -p 5432 -U replicator -D /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01 -Fp -Xs -P -R --checkpoint=fast --wal-method=stream --verbose
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/2000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1743757"
20146/30540 kB (65%), 0/1 tablespace (...er-01-standby-01/base/16384/2693)
30550/30550 kB (100%), 0/1 tablespace (...-01-standby-01/global/pg_control)
30550/30550 kB (100%), 1/1 tablespace                                         
pg_basebackup: write-ahead log end point: 0/2000120
pg_basebackup: waiting for background process to finish streaming ...
pg_basebackup: syncing data to disk ...
pg_basebackup: renaming backup_manifest.tmp to backup_manifest
pg_basebackup: base backup completed
[OK]   pg_basebackup complete
[INFO] Fixing postgresql.auto.conf to use IP-based primary_conninfo (matching golden backup)...
[2026-01-02 08:18:53 UTC] USER=www-data EUID=0 PID=1743766 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-02 08:18:54 UTC] USER=www-data EUID=0 PID=1743789 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-02 08:18:54 UTC] USER=www-data EUID=0 PID=1743810 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-02 08:18:54 UTC] USER=www-data EUID=0 PID=1743819 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[OK]   standby.signal verified and permissions set
[INFO] Fixing postgresql.conf with standby-specific settings...
[WARN] postgresql.conf not found at /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/postgresql.conf
[INFO] Verifying postgresql.auto.conf...
[WARN] postgresql.auto.conf not found - pg_basebackup may have failed
[2026-01-02 08:18:54 UTC] USER=www-data EUID=0 PID=1743842 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.UTx22n
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-02 08:18:54 UTC] USER=www-data EUID=0 PID=1743894 ACTION=fsop ARGS=cp /tmp/tmp.30uFKvCNTP /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/pg_hba.conf
[2026-01-02 08:18:54 UTC] USER=www-data EUID=0 PID=1743915 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/pg_hba.conf
[2026-01-02 08:18:54 UTC] USER=www-data EUID=0 PID=1743936 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01-standby-01.service
[2026-01-02 08:18:54 UTC] USER=www-data EUID=0 PID=1743961 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.vpl6HD /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01-standby-01.service
[2026-01-02 08:18:54 UTC] USER=www-data EUID=0 PID=1743986 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01-standby-01.service
[OK]   systemd unit written
[2026-01-02 08:18:55 UTC] USER=www-data EUID=0 PID=1744008 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 08:18:55 UTC] USER=www-data EUID=0 PID=1744029 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 08:18:55 UTC] USER=www-data EUID=0 PID=1744050 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-02 08:18:57 UTC] USER=www-data EUID=0 PID=1744171 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-worker-01-standby-01.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-02 08:18:58 UTC] USER=www-data EUID=0 PID=1744213 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-worker-01-standby-01.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Configuring synchronous replication on primary worker-01...
[INFO] Current synchronous_standby_names: ''
[INFO] Initializing synchronous_standby_names with first standby
[INFO] New synchronous_standby_names: 'ANY 1 (worker_01_standby_01)'
[2026-01-02 08:18:58 UTC] USER=www-data EUID=0 PID=1744281 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_commit = on;
ALTER SYSTEM
[2026-01-02 08:18:58 UTC] USER=www-data EUID=0 PID=1744305 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_standby_names = 'ANY 1 (worker_01_standby_01)';
ALTER SYSTEM
[2026-01-02 08:18:58 UTC] USER=www-data EUID=0 PID=1744328 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   ✅ Synchronous replication configured on primary
[OK]      Setting: ANY 1 (worker_01_standby_01)
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Skipping database/role provisioning on standby node (read-only)
[INFO]   Database/roles will be replicated from primary: worker-01
[INFO] Applying connection and memory optimizations...
[INFO] Standby will use primary's max_connections: 100
[INFO] Current settings: max_connections=100, work_mem=8MB
[INFO] Target settings (standby): max_connections=100, work_mem=8MB
[OK]   Connection settings already optimized
[INFO] Skipping password setting - this is a standby (read-only)
[INFO] Use primary's postgres password to connect to this standby
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] PostgreSQL IP: 10.100.1.211
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01-standby-01...
[INFO]   db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.211

[INFO]   ✅ db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.211    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com


[OK]   PostgreSQL 'identity-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key \
        host=db-identity-sau-main-dev-postgresql-worker-01-standby-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-worker-01-standby-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.211
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-worker-01-standby-01
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 8eaa8059-bede-4f71-ae1d-d26590a898da
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 📦 03 role...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 08:19:05 UTC] USER=www-data EUID=0 PID=1744699 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
⚠ This is a PostgreSQL STANDBY (read-only replica)
⚠ Skipping role creation - standby gets roles from primary via replication
⚠ Use the PRIMARY's credentials to connect to this standby


[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=worker-01-standby-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║                    IAM Database Schema Initialization                       ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] 🟢 Starting IAM schema provisioning...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

[INFO] 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: core
  Core Identity Directory (tenants, realms, identities, devices, MFA)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [1/20]: core/01-tenant
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "pgcrypto" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "citext" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
NOTICE:  schema "utils" already exists, skipping
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
NOTICE:  schema "core" already exists, skipping
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
NOTICE:  relation "tenant" already exists, skipping
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[OK] Table core/01-tenant initialized

[INFO] 🔸 Table [2/20]: core/02-realm
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  relation "realm" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_realm_keycloak_id" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_realm_tenant" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[OK] Table core/02-realm initialized

[INFO] 🔸 Table [3/20]: core/03-identity
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  relation "identity" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_unique_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_unique_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_type" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[OK] Table core/03-identity initialized

[INFO] 🔸 Table [4/20]: core/04-device
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  relation "device" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_device_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_fingerprint" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_trusted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_last_seen" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[OK] Table core/04-device initialized

[INFO] 🔸 Table [5/20]: core/05-identity_account
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  relation "identity_account" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_account_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_lockout" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_last_login" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[OK] Table core/05-identity_account initialized

[INFO] 🔸 Table [6/20]: core/06-identity_mfa
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  relation "identity_mfa" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_mfa_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_active" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[OK] Table core/06-identity_mfa initialized

[INFO] 🔸 Table [7/20]: core/07-external_idp_link
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  relation "external_idp_link" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_external_idp_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_provider" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_email" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[OK] Table core/07-external_idp_link initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: policy
  RBAC/ABAC Authorization (clients, roles, permissions, policies)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [8/20]: policy/01-client
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
NOTICE:  schema "policy" already exists, skipping
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  relation "client" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_client_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_key" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_status" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[OK] Table policy/01-client initialized

[INFO] 🔸 Table [9/20]: policy/02-resource
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  relation "resource" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_resource_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_external" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_owner" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[OK] Table policy/02-resource initialized

[INFO] 🔸 Table [10/20]: policy/03-scope
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  relation "scope" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_scope_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_scope_name" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[OK] Table policy/03-scope initialized

[INFO] 🔸 Table [11/20]: policy/04-permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  relation "permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_permission_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_resource" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[OK] Table policy/04-permission initialized

[INFO] 🔸 Table [12/20]: policy/05-role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  relation "role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_keycloak" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[OK] Table policy/05-role initialized

[INFO] 🔸 Table [13/20]: policy/06-role_permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  relation "role_permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_permission_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_permission_perm" already exists, skipping
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[OK] Table policy/06-role_permission initialized

[INFO] 🔸 Table [14/20]: policy/07-identity_role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  relation "identity_role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_role_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_active" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[OK] Table policy/07-identity_role initialized

[INFO] 🔸 Table [15/20]: policy/08-policy_rule
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  relation "policy_rule" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_policy_rule_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_enabled" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_priority" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[OK] Table policy/08-policy_rule initialized

[INFO] 🔸 Table [16/20]: policy/09-api_key
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  relation "api_key" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_api_key_prefix" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[OK] Table policy/09-api_key initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: audit
  Audit & Risk Logging (auth events, admin actions, risk decisions)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [17/20]: audit/01-auth_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
NOTICE:  schema "audit" already exists, skipping
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
NOTICE:  relation "auth_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_auth_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_result" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_ip" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_session" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_trace" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_risk" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[OK] Table audit/01-auth_event initialized

[INFO] 🔸 Table [18/20]: audit/02-admin_action
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
NOTICE:  relation "admin_action" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_admin_action_actor" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_target" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_trace" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[OK] Table audit/02-admin_action initialized

[INFO] 🔸 Table [19/20]: audit/03-risk_decision
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
NOTICE:  relation "risk_decision" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_risk_decision_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_level" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_decision" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_auth" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[OK] Table audit/03-risk_decision initialized

[INFO] 🔸 Table [20/20]: audit/04-consent_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
NOTICE:  relation "consent_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_consent_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_version" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_granted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  relation "audit.auth_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  relation "audit.auth_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  relation "audit.admin_action_2026_03" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  relation "audit.admin_action_2026_04" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  relation "audit.risk_decision_2026_03" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  relation "audit.risk_decision_2026_04" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  relation "audit.consent_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  relation "audit.consent_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[OK] Table audit/04-consent_event initialized


════════════════════════════════════════════════════════════════════════════
[OK] ✅ IAM Schema Initialization Complete!
[OK] All 20 tables initialized successfully

Schemas created:
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

Design highlights:
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

════════════════════════════════════════════════════════════════════════════

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=worker-01-standby-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: identity/run.sh started for SERVICE=identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity
🔍 DEBUG_CHECKPOINT_A3: SERVICE_ROOT exists, discovering table folders
🔍 DEBUG_CHECKPOINT_A4: Found subfolder: auth
🔍 DEBUG_CHECKPOINT_A4b: Checking for nested schema layout in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth
🔍 DEBUG_CHECKPOINT_A4c: Found nested steps dir: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps (display: auth/login)
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: auth/login|/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
[INFO] 📚 Detected grouped table folders under identity/: auth/login

🔍 DEBUG_CHECKPOINT_A7: Current IDENTIFIER=coordinator
🔍 DEBUG_CHECKPOINT_A8_PROCEED: Processing tables on coordinator/main node
🔍 DEBUG_CHECKPOINT_A9: Processing table: auth/login at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
[INFO] 🔸 Table group: auth/login
🔍 DEBUG_CHECKPOINT_A10: About to run numbered steps for table: auth/login
🔍 DEBUG_CHECKPOINT_B1: run_all_numbered_steps_in_dir called for dir=/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps table=auth/login
🔍 DEBUG_CHECKPOINT_B2: Found 1 numbered steps: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B3: About to run step: 01-init-schema.sh
Ab substep 0 compelete start
[DEBUG] Tracking substep start: steps/01-install/steps/identity/auth/login/01-init-schema (RUN_UUID=58d74c86-e962-4adb-a920-46eae94b25e4)
Ab substep 0 compelete start
[INFO] 📦 01 init schema...
Ab substep 1 compelete start
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing auth.login_account table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating auth schema...
NOTICE:  schema "auth" already exists, skipping
CREATE SCHEMA
✅ Schema created
🔧 Creating account_status ENUM...
DO
✅ ENUM created
🔧 Creating auth.login_account table...
NOTICE:  relation "login_account" already exists, skipping
CREATE TABLE
✅ Table created (Citus-compatible with region_hint in all constraints)
🔧 Creating indexes...
NOTICE:  relation "idx_login_account_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_login_account_username" already exists, skipping
CREATE INDEX
✅ Indexes created
ℹ️  Table already registered with Citus
🎉 Schema initialization complete for fastorder_identity_sau_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

📊 Registering environment in monitoring database (obs schema)...
   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
   Resource IP: 142.93.238.16
⚠️  Could not connect to monitoring database, skipping registration
   You can manually register later using:
   /opt/fastorder/bash/scripts/env_app_setup/setup/04-postgresql/steps/register-authN-af-aaaa1-dev.sh

==========================================
✅ Schema initialization complete!
==========================================
Ab substep 1 compelete end
Ab substep 2 compelete start
Ab substep 2 compelete end

🔍 DEBUG_CHECKPOINT_B4: Completed step: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B5: All numbered steps completed for /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A11: Completed numbered steps for table: auth/login
compeleted here

🔍 DEBUG_CHECKPOINT_A12: All tables processed
End of 04-postgresql/steps/01-install/steps/identity/run.sh

✓ ✅ Standby worker-01-standby-01 setup completed

✓ ✅ PostgreSQL installation completed
[INFO] Discovering additional setup steps...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 02-pg-bouncer.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up PgBouncer connection pooling...
[2026-01-02 08:20:43 UTC] USER=www-data EUID=0 PID=1748322 ACTION=fsop ARGS=rm -f /tmp/pgbouncer-ip.service /tmp/pgbouncer.service
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] Checking for existing PgBouncer application environment in topology …
[OK]   Using existing PgBouncer environment:
[INFO]   IP:     10.100.1.204
[INFO]   FQDN:   db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Domain: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO] Ensuring /etc/hosts entry for db-identity-sau-main-dev-postgresql-bouncer.fastorder.com …
[OK]   /etc/hosts already contains entry for db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[WARN] IP 10.100.1.204 is assigned to multiple interfaces:
    inet 10.100.1.103/32 scope global lo
       valid_lft forever preferred_lft forever
    inet 10.100.1.204/32 scope global lo:pgbouncer
--
    inet 10.100.1.214/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.100.1.204/32 scope global eth0:pgbouncer
[WARN] This may cause routing issues
[INFO] Final verification of /etc/hosts entry for db-identity-sau-main-dev-postgresql-bouncer.fastorder.com …
[OK]   /etc/hosts correctly maps db-identity-sau-main-dev-postgresql-bouncer.fastorder.com to 10.100.1.204
[OK]   PgBouncer IP 10.100.1.204 already correctly bound to lo:pgbouncer
[2026-01-02 08:20:44 UTC] USER=www-data EUID=0 PID=1748408 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 08:20:46 UTC] USER=www-data EUID=0 PID=1748502 ACTION=passthru ARGS=systemctl restart pgbouncer-ip@identity-sau-main-dev.service
[2026-01-02 08:20:46 UTC] USER=www-data EUID=0 PID=1748513 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer-ip@identity-sau-main-dev.service
[OK]   pgbouncer-ip@identity-sau-main-dev.service is active
[2026-01-02 08:20:46 UTC] USER=www-data EUID=0 PID=1748537 ACTION=fsop ARGS=mkdir -p /etc/pgbouncer/identity-sau-main-dev
[2026-01-02 08:20:46 UTC] USER=www-data EUID=0 PID=1748546 ACTION=fsop ARGS=mkdir -p /run/pgbouncer/identity-sau-main-dev
[2026-01-02 08:20:46 UTC] USER=www-data EUID=0 PID=1748555 ACTION=fsop ARGS=mkdir -p /var/log/pgbouncer/identity-sau-main-dev
[2026-01-02 08:20:46 UTC] USER=www-data EUID=0 PID=1748564 ACTION=fsop ARGS=chmod 750 /etc/pgbouncer/identity-sau-main-dev
[2026-01-02 08:20:46 UTC] USER=www-data EUID=0 PID=1748573 ACTION=fsop ARGS=chmod 750 /run/pgbouncer/identity-sau-main-dev
[2026-01-02 08:20:46 UTC] USER=www-data EUID=0 PID=1748582 ACTION=fsop ARGS=chmod 750 /var/log/pgbouncer/identity-sau-main-dev
[2026-01-02 08:20:46 UTC] USER=www-data EUID=0 PID=1748591 ACTION=fsop ARGS=chown root:postgres /etc/pgbouncer/identity-sau-main-dev
[2026-01-02 08:20:46 UTC] USER=www-data EUID=0 PID=1748600 ACTION=fsop ARGS=chown postgres:postgres /run/pgbouncer/identity-sau-main-dev
[2026-01-02 08:20:46 UTC] USER=www-data EUID=0 PID=1748609 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbouncer/identity-sau-main-dev
[INFO] Generating pgbouncer_admin client certificates...
[INFO] ⏳ This may take 30-60 seconds...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    pgbouncer_admin
Identifier:  pgbouncer
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        pgbouncer
  User (CN):   pgbouncer_admin
  Hostname:    db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:20:47 UTC] USER=www-data EUID=0 PID=1748647 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-pgbouncer-pgbouncer_admin
[2026-01-02 08:20:47 UTC] USER=www-data EUID=0 PID=1748656 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-02 08:20:47 UTC] USER=www-data EUID=0 PID=1748665 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
[2026-01-02 08:20:47 UTC] USER=www-data EUID=0 PID=1748674 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-02 08:20:47 UTC] USER=www-data EUID=0 PID=1748683 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = pgbouncer_admin
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:48 UTC] USER=www-data EUID=0 PID=1748702 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:48 UTC] USER=www-data EUID=0 PID=1748711 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:48 UTC] USER=www-data EUID=0 PID=1748720 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 08:20:48 UTC] USER=www-data EUID=0 PID=1748729 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 08:20:48 UTC] USER=www-data EUID=0 PID=1748738 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 08:20:48 UTC] USER=www-data EUID=0 PID=1748747 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 08:20:48 UTC] USER=www-data EUID=0 PID=1748756 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 08:20:48 UTC] USER=www-data EUID=0 PID=1748765 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 08:20:48 UTC] USER=www-data EUID=0 PID=1748774 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 08:20:48 UTC] USER=www-data EUID=0 PID=1748783 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 08:20:48 UTC] USER=www-data EUID=0 PID=1748792 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 08:20:48 UTC] USER=www-data EUID=0 PID=1748801 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 08:20:48 UTC] USER=www-data EUID=0 PID=1748810 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:48 UTC] USER=www-data EUID=0 PID=1748819 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 08:20:48 UTC] USER=www-data EUID=0 PID=1748828 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1748839 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1748848 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1748857 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1748883 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1748892 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1748901 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1748910 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1748919 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1748928 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1748938 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1748948 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1748959 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1748968 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1748977 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1748987 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1748997 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1749006 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1749015 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1749024 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1749033 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1749042 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1749051 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1749060 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1749069 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1749078 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 08:20:49 UTC] USER=www-data EUID=0 PID=1749087 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749097 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749107 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749116 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749125 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749134 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749143 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749152 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749161 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749170 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749179 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749188 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749197 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749207 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749217 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749226 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749235 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749244 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749253 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749262 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749271 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749280 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749289 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749298 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749307 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749317 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749327 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749336 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749345 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749354 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:20:50 UTC] USER=www-data EUID=0 PID=1749363 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749374 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749383 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749392 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749401 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: pgbouncer_admin
Node: pgbouncer
FQDN: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-bouncer.fastorder.com -U pgbouncer_admin -d postgres

[OK]   mTLS client certificate present: /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[INFO] Creating symlinks to canonical certificates in /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend...
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749417 ACTION=fsop ARGS=mkdir -p /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749426 ACTION=fsop ARGS=mkdir -p /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749436 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749445 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749454 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt
[INFO] Creating coordinator CA symlink for PostgreSQL server verification...
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749463 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] Verifying canonical certificate permissions...
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749472 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749483 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749492 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749501 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[OK]   Backend certificate symlinks created in /etc/ssl
[OK]   Coordinator CA symlink created for server verification
[OK]   Certificates already in canonical location - no symlinks needed
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749512 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.crt
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749521 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.key
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749530 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749540 ACTION=fsop ARGS=test -r /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] PgBouncer will use PostgreSQL coordinator CA: /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[OK]   PostgreSQL coordinator at db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432 is reachable
[INFO] Dumping SCRAM secrets from coordinator for PgBouncer auth_file …
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749559 ACTION=fsop ARGS=cp /tmp/tmp.PIiu2WdqGZ /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-02 08:20:51 UTC] USER=www-data EUID=0 PID=1749569 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-02 08:20:52 UTC] USER=www-data EUID=0 PID=1749578 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[OK]   Auth file written: /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[INFO] Generated new password for pgbouncer_admin
[INFO] Ensuring PgBouncer admin role 'pgbouncer_admin' exists in Postgres (coordinator) …
[OK]   Role pgbouncer_admin created/updated successfully
[SECRETS] Setting credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/pgbouncer_admin
✓ [SECRETS] Credentials created in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/pgbouncer_admin
[INFO] ✅ PgBouncer admin password stored in centralized secrets vault
[INFO] Re-fetching SCRAM secrets after role creation to ensure pgbouncer_admin is included …
[2026-01-02 08:20:59 UTC] USER=www-data EUID=0 PID=1749758 ACTION=fsop ARGS=cp /tmp/tmp.zDuBh7CoJd /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-02 08:20:59 UTC] USER=www-data EUID=0 PID=1749767 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-02 08:20:59 UTC] USER=www-data EUID=0 PID=1749776 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[OK]   Auth file updated with pgbouncer_admin SCRAM hash
[INFO] Auth file contains [2026-01-02 08:20:59 UTC] USER=www-data EUID=0 PID=1749786 ACTION=passthru ARGS=bash -c wc -l < '/etc/pgbouncer/identity-sau-main-dev/userlist.txt'
4 user(s)
[OK]   Admin 'pgbouncer_admin' password generated and saved
[INFO] Configuring PostgreSQL to prevent Citus metadata sync hangs...
ALTER ROLE
[OK]   Disabled Citus metadata sync for pgbouncer_admin
[INFO] Verifying application database fastorder_identity_sau_main_dev_db exists...
[OK]   ✓ Database fastorder_identity_sau_main_dev_db exists
[INFO] Granting permissions to pgbouncer_admin on fastorder_identity_sau_main_dev_db...
GRANT
[OK]   ✓ Granted CONNECT on fastorder_identity_sau_main_dev_db to pgbouncer_admin
GRANT
[OK]   ✓ Granted USAGE on schema public to pgbouncer_admin
GRANT
[OK]   ✓ Granted SELECT on all tables to pgbouncer_admin
ALTER DATABASE
[OK]   Set synchronous_commit=local for fastorder_identity_sau_main_dev_db
[INFO] Ensuring pg_hba.conf entry for pgbouncer_admin …
[INFO] Adding pg_hba.conf entries for pgbouncer_admin with cert auth …
[2026-01-02 08:21:00 UTC] USER=unknown EUID=33 PID=1749823 ACTION=-u ARGS=postgres bash
ERROR: Invalid or unauthorized action: -u
[OK]   pg_hba.conf updated and PostgreSQL configuration reloaded
[WARN] pg_hba.conf entry may not have loaded correctly
[INFO] Writing /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini …
[2026-01-02 08:21:01 UTC] USER=www-data EUID=0 PID=1749848 ACTION=fsop ARGS=cp /tmp/tmp.9oIH69qOVG /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
[2026-01-02 08:21:01 UTC] USER=www-data EUID=0 PID=1749857 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
[2026-01-02 08:21:01 UTC] USER=www-data EUID=0 PID=1749866 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
[2026-01-02 08:21:02 UTC] USER=www-data EUID=0 PID=1749890 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbouncer/identity-sau-main-dev /run/pgbouncer/identity-sau-main-dev /var/log/pgbouncer/identity-sau-main-dev
[2026-01-02 08:21:02 UTC] USER=www-data EUID=0 PID=1749899 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[OK]   pgbouncer.ini ready
[INFO] Verifying TLS settings in pgbouncer.ini:
[2026-01-02 08:21:02 UTC] USER=www-data EUID=0 PID=1749909 ACTION=fsop ARGS=grep -E (client_tls_sslmode|server_tls) /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
[INFO] Verifying PgBouncer server certificate files:
[2026-01-02 08:21:02 UTC] USER=www-data EUID=0 PID=1749918 ACTION=fsop ARGS=test -r /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[OK]   Server cert readable by postgres: /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-01-02 08:21:02 UTC] USER=www-data EUID=0 PID=1749927 ACTION=fsop ARGS=test -r /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[OK]   Server key readable by postgres: /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[INFO] Verifying coordinator CA certificate:
[2026-01-02 08:21:02 UTC] USER=www-data EUID=0 PID=1749936 ACTION=fsop ARGS=test -r /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[OK]   Coordinator CA readable by postgres: /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] Preflight: stopping any conflicting PgBouncer on 6432 …
[2026-01-02 08:21:02 UTC] USER=www-data EUID=0 PID=1749945 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer.service
[2026-01-02 08:21:02 UTC] USER=www-data EUID=0 PID=1749955 ACTION=passthru ARGS=systemctl stop pgbouncer@identity-sau-main-dev.service
Failed to stop pgbouncer@identity-sau-main-dev.service: Unit pgbouncer@identity-sau-main-dev.service not loaded.
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json?all=1": dial unix /var/run/docker.sock: connect: permission denied
[WARN] Killing existing pgbouncer processes: 1421884
1425552
[2026-01-02 08:21:02 UTC] USER=www-data EUID=0 PID=1749980 ACTION=passthru ARGS=bash -c kill -9 1421884
[2026-01-02 08:21:02 UTC] USER=www-data EUID=0 PID=1749991 ACTION=passthru ARGS=bash -c kill -9 1425552
[2026-01-02 08:21:04 UTC] USER=www-data EUID=0 PID=1750028 ACTION=passthru ARGS=systemctl daemon-reload
[OK]   systemd unit installed: pgbouncer@identity-sau-main-dev.service
[INFO] Running pre-flight IP conflict check for 10.100.1.204:6432 …
[WARN] IP conflict checker not found at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/check-ip-conflicts.sh
[WARN] Skipping pre-flight check - conflicts may occur
[INFO] Starting PgBouncer (identity-sau-main-dev) …
[2026-01-02 08:21:06 UTC] USER=www-data EUID=0 PID=1750128 ACTION=passthru ARGS=systemctl restart pgbouncer@identity-sau-main-dev.service
[2026-01-02 08:21:06 UTC] USER=www-data EUID=0 PID=1750138 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer@identity-sau-main-dev.service
[OK]   Service ACTIVE
[INFO] Verifying auth_file before probing …
[INFO] Auth file contains 4 user(s)
[WARN] Auth file does NOT contain pgbouncer_admin entry - authentication will fail
[INFO] Probing admin console via SSL (psql to database 'pgbouncer') …
[INFO] Retrieved password from vault for admin console probe
[WARN] SSL connection issue detected
[INFO] Attempting connection with sslmode=disable for testing...
[WARN] If this fails, check PgBouncer client_tls_sslmode setting
[WARN] Admin console probe failed (see error below)
psql: error: connection to server at "10.100.1.204", port 6432 failed: SSL error: certificate verify failed
[WARN] Troubleshooting:
[WARN]   1. Check auth_file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[WARN]   2. Test with: PGPASSWORD='kppzNMG6WDrJWGUYcBARr4ME' psql -h 10.100.1.204 -p 6432 -U pgbouncer_admin -d pgbouncer
[WARN]   3. Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@identity-sau-main-dev.service -n 50

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Running Comprehensive PgBouncer Verification Tests
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Password extracted: kppzNMG6WD... (using postgres user certificates)

[INFO] Test 1/7: Admin Console - SHOW POOLS
[WARN] ✗ SHOW POOLS: FAILED
[WARN] Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@identity-sau-main-dev.service -n 50

[INFO] Test 2/7: Admin Console - SHOW VERSION
[WARN] ✗ SHOW VERSION: FAILED

[INFO] Test 3/7: Admin Console - SHOW STATS
[WARN] ✗ SHOW STATS: FAILED

[INFO] Test 4/7: Admin Console - SHOW DATABASES
[WARN] ✗ SHOW DATABASES: FAILED

[INFO] Test 5/7: Admin Console - SHOW CONFIG
[WARN] ✗ SHOW CONFIG: FAILED
psql   "host=db-identity-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_identity_sau_main_dev_db user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME    connect_timeout=5 sslmode=verify-full    sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt    sslcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt    sslkey=/etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key"   --no-psqlrc -Atc 'SELECT version();'

[INFO] Test 6/7: Application Database - SELECT version()
[WARN] ✗ Application database query: FAILED (timeout or connection issue)
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[INFO] Test 7/8: Application Database - Connection Details
[WARN] ✗ Connection details: FAILED (timeout or connection issue)
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[INFO] Test 8/8: End-to-End Application Routing - Pool Verification
[INFO]   Running actual queries through PgBouncer to verify routing and pooling...
[WARN] ✗ End-to-end routing verification: FAILED - All 3 queries failed
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[WARN]    Otherwise check if database fastorder_identity_sau_main_dev_db exists and user pgbouncer_admin has permissions

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Verification Complete - Tests 1-5 PASSED (Admin console verified)
[WARN]   Tests 6-8 FAILED - Application database not accessible
[WARN]   This is expected if Citus is not set up yet
[WARN]   Run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[OK]   PgBouncer is up for identity-sau-main-dev

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Connection Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Password stored in: AWS Secrets Manager (fastorder/db/web/ksa/main/dev/postgresqlidentity/sau/main/dev/coordinator-pgbouncer_admin)
Current password: kppzNMG6WDrJWGUYcBARr4ME

1. Admin Console (using IP address to avoid DNS/SSL issues):
   psql "host=10.100.1.204 port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME sslmode=verify-full sslrootcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

2. Admin Console (using hostname):
   psql "host=db-identity-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME sslmode=verify-full sslrootcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

3. Application Database:
   psql "host=db-identity-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_identity_sau_main_dev_db sslkey=/etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME sslmode=verify-full sslrootcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

4. Using .pgpass file:
   echo "db-identity-sau-main-dev-postgresql-bouncer.fastorder.com:6432:*:pgbouncer_admin:kppzNMG6WDrJWGUYcBARr4ME" >> ~/.pgpass
   chmod 600 ~/.pgpass
   psql -h db-identity-sau-main-dev-postgresql-bouncer.fastorder.com -p 6432 -U pgbouncer_admin -d fastorder_identity_sau_main_dev_db

5. Retrieve password from vault:
   source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
   PGPASSWORD="$(get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password')" \
     psql -h 10.100.1.204 -p 6432 -U pgbouncer_admin -d pgbouncer -c "SHOW POOLS;"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Architecture
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  • Default db 'fastorder_identity_sau_main_dev_db' → Citus coordinator (db-identity-sau-main-dev-postgresql-coordinator.fastorder.com)
  • Worker access: 'fastorder_identity_sau_main_dev_db_worker_1', 'fastorder_identity_sau_main_dev_db_worker_2', … (if exist)
  • Client TLS: require (password auth) / verify-full (mTLS with certs)
  • Server TLS: verify-full (PgBouncer validates PostgreSQL certs)
  • Auth: SCRAM-SHA-256 via /etc/pgbouncer/identity-sau-main-dev/userlist.txt
  • Pool mode: transaction (stateless connections)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Management
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Service Status:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer@identity-sau-main-dev.service
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@identity-sau-main-dev.service

Logs:
  command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@identity-sau-main-dev.service -f
  /usr/local/bin/fastorder-provisioning-wrapper.sh tail -f /var/log/pgbouncer/identity-sau-main-dev/pgbouncer.log

Reload Config:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@identity-sau-main-dev.service

Restart:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@identity-sau-main-dev.service

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Files
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Config:        /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
Auth file:     /etc/pgbouncer/identity-sau-main-dev/userlist.txt
Server cert:   /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.crt
Server key:    /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.key
CA cert:       /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt
PG CA:         /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
Logs:          /var/log/pgbouncer/identity-sau-main-dev/pgbouncer.log

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Troubleshooting
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


If "SASL authentication failed":
  1. Check auth file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/identity-sau-main-dev/userlist.txt
  2. Verify pgbouncer_admin is present with SCRAM hash
  3. Get password from vault:
     source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
     get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password'
  4. Reload PgBouncer: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@identity-sau-main-dev.service

If "no pg_hba.conf entry":
  1. Check pg_hba.conf on coordinator
  2. Add rule: hostssl all pgbouncer_admin 10.100.1.204/32 cert clientcert=verify-full
  3. Reload PostgreSQL

To add users to PgBouncer:
  1. Create user in PostgreSQL with password
  2. Re-run SCRAM dump:
     psql "host=db-identity-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 dbname=postgres user=postgres \
       sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt \
       sslcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt sslkey=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key" \
       -Atc "SELECT '\"' || rolname || '\" \"' || rolpassword || '\"' \
             FROM pg_authid WHERE rolpassword LIKE 'SCRAM-SHA-256%' \
             AND rolcanlogin ORDER BY rolname;" | command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop tee /etc/pgbouncer/identity-sau-main-dev/userlist.txt
  3. Reload: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@identity-sau-main-dev.service

[INFO] Registering PgBouncer node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        identity-sau-main-dev-pgbouncer
[INFO]   Identifier Parent: postgresql
[INFO]   IP:                10.100.1.204
[INFO]   Port:              6432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 426480a5-2f64-4fc0-b2b5-710f9ccb059a
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   PgBouncer node registered to observability API
✓ ✅ PgBouncer setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 03-citus-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] CITUS DISTRIBUTED CLUSTER SETUP
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Phase 1: Installing Citus extension on workers...
[INFO] Phase 2: Setting up coordinator and registering workers...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 📦 PHASE 1: Installing Citus extension on 1 worker(s)...

[INFO] → Worker 1/1: Installing Citus on worker-01...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS CLUSTER SETUP
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔧 Setting up Citus Worker...
[INFO] Temporarily disabling synchronous replication for extension installation...
t
[INFO] Installing Citus extension on worker...
[OK]   Citus extension installed on worker
[INFO] Restoring synchronous replication settings...
t
[INFO] Worker Citus extension installed - registration will happen when coordinator setup runs

[OK]   Citus setup complete for worker-01
[INFO] ═══════════════════════════════════════════════════════════════════════════════
✓   ✅ Citus extension installed on worker-01

✓ ✅ Phase 1 Complete: All 1 workers have Citus extension installed

[INFO] 🔧 PHASE 2: Setting up Citus coordinator and registering workers...

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS CLUSTER SETUP
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔧 Setting up Citus Coordinator...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] DIAGNOSTIC: Configuration Variables
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PG_WORKERS_NUM: 1
[INFO] ENV_ID: identity-sau-main-dev
[INFO] DOMAIN: fastorder.com
[INFO] PORT: 5432
[INFO] SOCKET_DIR: /var/run/postgresql-identity-sau-main-dev-coordinator
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Ensuring postgres client certificates exist for coordinator...
[OK]   Postgres client certificates already exist for coordinator
[INFO] Adding citus_cert_map to coordinator pg_ident.conf...
[OK]   pg_ident.conf updated for coordinator
[INFO] Installing Citus extension on coordinator...
[OK]   Citus extension installed on coordinator (postgres database)
[INFO] Installing Citus extension on application database: fastorder_identity_sau_main_dev_db...
[OK]   Citus extension installed on application database: fastorder_identity_sau_main_dev_db
[INFO] Configuring Citus SSL connection parameters...
[2026-01-02 08:21:22 UTC] USER=www-data EUID=0 PID=1750606 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
[OK]   ✅ Citus SSL connection parameters configured: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[WARN] Node not identified as coordinator, initializing...
[INFO] Checking coordinator configuration...
[INFO] Persisting citus.local_hostname to postgresql.conf...
[2026-01-02 08:21:24 UTC] USER=www-data EUID=0 PID=1750653 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /var/lib/postgresql/17/identity-sau-main-dev/coordinator/postgresql.conf
[2026-01-02 08:21:24 UTC] USER=www-data EUID=0 PID=1750674 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
[OK]   ✅ citus.local_hostname persisted to config and reloaded
[INFO] Configuring coordinator hostname in postgres database: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432

[OK]   ✅ Coordinator hostname set to db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432 in postgres database
[INFO] Checking coordinator configuration in application database: fastorder_identity_sau_main_dev_db...
[WARN] ⚠️  Coordinator registered as 'localhost' in application database, fixing...
[INFO] Configuring coordinator hostname in application database: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432
[OK]   ✅ Coordinator hostname set to db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432 in application database
[INFO] Validating coordinator configuration before worker registration...
[OK]   ✅ Coordinator hostname validated: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[OK]   ✅ citus_tables view is accessible
[INFO] Checking coordinator self-registration...
[OK]   ✅ Coordinator is already self-registered
[INFO] Configuring coordinator shard placement policy...
[OK]   ✅ Coordinator already configured in postgres database (shouldhaveshards = false)
[WARN] ⚠️  Coordinator has 17 shards in fastorder_identity_sau_main_dev_db - cannot set shouldhaveshards=false
[WARN]    You must rebalance shards to workers first, then run this setup again
[WARN]    Skipping shouldhaveshards configuration for application database
[INFO] Registering 1 worker(s) to Citus cluster...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PRE-FLIGHT: Checking worker availability...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking worker worker-01...
[INFO]   FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[OK]   ✅ Worker worker-01 is reachable via SSL
[OK]   All workers are reachable - proceeding with registration

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Adding Citus worker: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Adding citus_cert_map to worker-01 pg_ident.conf...
[OK]   pg_ident.conf updated for worker-01
[INFO] Configuring worker worker-01 HBA for coordinator (10.100.1.213) access...
[OK]   Worker worker-01 HBA configured for coordinator (10.100.1.213)
[INFO] Adding replication rules for 3 standby(s)...
[OK]   Replication rules already exist for worker-01
[INFO] Reloading worker worker-01 to apply HBA changes...
[2026-01-02 08:21:28 UTC] USER=www-data EUID=0 PID=1750811 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
[INFO] Configuring coordinator HBA for worker worker-01 (10.100.1.214) access...
[OK]   Coordinator HBA configured for worker worker-01 (10.100.1.214)
[INFO] Reloading coordinator to apply HBA changes...
[2026-01-02 08:21:28 UTC] USER=www-data EUID=0 PID=1750841 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
[INFO] Ensuring postgres client certificates exist for worker-01...
[OK]   Postgres client certificates already exist for worker-01
[INFO] Configuring citus.node_conninfo on worker-01...
[2026-01-02 08:21:29 UTC] USER=www-data EUID=0 PID=1750860 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
[OK]   citus.node_conninfo configured on worker-01
[INFO] Temporarily relaxing sync-rep on worker worker-01...
t
[OK]   Worker worker-01 sync-rep relaxed (was: sync_commit=on)
[INFO] Ensuring Citus extension on worker databases...
CREATE EXTENSION
CREATE EXTENSION
[INFO] Running citus_add_node with 180s timeout...
NOTICE:  shards are still on the coordinator after adding the new node
HINT:  Use SELECT rebalance_table_shards(); to balance shards data between workers and coordinator or SELECT citus_drain_node('db-identity-sau-main-dev-postgresql-coordinator.fastorder.com',5432); to permanently move shards away from the coordinator.
2
[INFO] Restoring worker worker-01 sync-rep settings...
t
[OK]   Worker worker-01 sync-rep restored
[OK]   ✅ Worker db-identity-sau-main-dev-postgresql-worker-01.fastorder.com successfully added to Citus cluster
[INFO]    Node ID: 2
[INFO]    Registered in: postgres, fastorder_identity_sau_main_dev_db
[OK]   Worker worker-01 registration successful
[INFO] Configuring worker worker-01 shard placement policy...
[OK]   ✅ Worker worker-01 configured to hold shards in all databases


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] POST-REGISTRATION: Verifying cluster state...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Expected workers: 1
[INFO] Registered workers: 1
[OK]   ✅ All 1 workers successfully registered!

[INFO] Citus cluster configuration:
db-identity-sau-main-dev-postgresql-coordinator.fastorder.com  5432  0  t  primary  f
db-identity-sau-main-dev-postgresql-worker-01.fastorder.com    5432  1  t  primary  t

[INFO] Note: groupid=0 is the coordinator, groupid>0 are workers
[INFO]       shouldhaveshards: false=query router only, true=holds data shards

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] FINAL VALIDATION: Verifying configuration persistence...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:21:33 UTC] USER=www-data EUID=0 PID=1751132 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /var/lib/postgresql/17/identity-sau-main-dev/coordinator/postgresql.conf
[OK]   ✅ citus.local_hostname persisted in postgresql.conf
[OK]   ✅ All 1 worker(s) successfully registered and verified

[OK]   ✅ All validation checks passed
[OK]   Citus coordinator setup complete

[OK]   Citus setup complete for coordinator
[INFO] ═══════════════════════════════════════════════════════════════════════════════

✓ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ CITUS CLUSTER SETUP COMPLETED SUCCESSFULLY
✓    Coordinator: Ready and accepting connections
✓    Workers registered: 1
✓ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 05-backup-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up coordinator backup...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for identity-sau-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-01-02 08:21:35 UTC] USER=www-data EUID=0 PID=1751200 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-02 08:21:35 UTC] USER=www-data EUID=0 PID=1751209 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/identity-sau-main-dev
[2026-01-02 08:21:35 UTC] USER=www-data EUID=0 PID=1751218 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-01-02 08:21:35 UTC] USER=www-data EUID=0 PID=1751227 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-02 08:21:35 UTC] USER=www-data EUID=0 PID=1751236 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-02 08:21:35 UTC] USER=www-data EUID=0 PID=1751245 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-02 08:21:43 UTC] USER=www-data EUID=0 PID=1751310 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-02 08:21:44 UTC] USER=www-data EUID=0 PID=1751319 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-02 08:21:44 UTC] USER=www-data EUID=0 PID=1751328 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-01-02 08:21:44 UTC] USER=www-data EUID=0 PID=1751337 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-02 08:21:44 UTC] USER=www-data EUID=0 PID=1751346 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-identity-sau-main-dev
[2026-01-02 08:21:44 UTC] USER=www-data EUID=0 PID=1751367 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-02 08:21:44 UTC] USER=www-data EUID=0 PID=1751376 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-01-02 08:21:44 UTC] USER=www-data EUID=0 PID=1751385 ACTION=fsop ARGS=find /var/lib/postgresql/17/identity-sau-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-01-02 08:21:44 UTC] USER=www-data EUID=0 PID=1751394 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-02 08:21:44 UTC] USER=www-data EUID=0 PID=1751405 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-01-02 08:21:44 UTC] USER=www-data EUID=0 PID=1751414 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-01-02 08:21:44 UTC] USER=www-data EUID=0 PID=1751423 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-02 08:21:44 UTC] USER=www-data EUID=0 PID=1751432 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/PG_VERSION
[2026-01-02 08:21:44 UTC] USER=www-data EUID=0 PID=1751442 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza identity-sau-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-02 08:21:45 UTC] USER=www-data EUID=0 PID=1751496 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-02 08:21:47 UTC] USER=www-data EUID=0 PID=1751518 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 08:21:51 UTC] USER=www-data EUID=0 PID=1751567 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-02 08:21:51 UTC] USER=www-data EUID=0 PID=1751591 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator --log-level-console=info check
2026-01-02 08:21:51.983 P00   INFO: check command begin 2.56.0: --exec-id=1751598-8f984634 --log-level-console=info --log-level-file=debug --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-02 08:21:52.038 P00   INFO: check repo1 configuration (primary)
2026-01-02 08:21:52.063 P00  ERROR: [028]: backup and archive info files exist but do not match the database
                                    HINT: is this the correct stanza?
                                    HINT: did an error occur during stanza-upgrade?
2026-01-02 08:21:52.063 P00   INFO: check command end: aborted with exception [028]
[WARN] ⚠️  Stanza verification failed - this may be normal if WAL archiving hasn't started yet
[WARN]    The backup system is configured and will work once WAL segments are generated

[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-02 08:21:52 UTC] USER=www-data EUID=0 PID=1751612 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-02 08:21:52 UTC] USER=www-data EUID=0 PID=1751621 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-02 08:21:52 UTC] USER=www-data EUID=0 PID=1751639 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[2026-01-02 08:21:52 UTC] USER=www-data EUID=0 PID=1751648 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-02 08:21:52 UTC] USER=www-data EUID=0 PID=1751668 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-identity-sau-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-01-02 08:21:52 UTC] USER=www-data EUID=0 PID=1751686 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 08:21:52 UTC] USER=www-data EUID=0 PID=1751695 ACTION=fsop ARGS=sed -i s|__ENV_ID__|identity-sau-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 08:21:52 UTC] USER=www-data EUID=0 PID=1751708 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/var/lib/postgresql/17/identity-sau-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 08:21:52 UTC] USER=www-data EUID=0 PID=1751730 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 08:21:52 UTC] USER=www-data EUID=0 PID=1751755 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-02 08:21:52.906 P00   INFO: start command begin 2.56.0: --exec-id=1751845-dabe4ecf --log-level-console=info --log-level-file=debug --stanza=identity-sau-main-dev-coordinator
2026-01-02 08:21:52.907 P00   WARN: stop file does not exist for stanza identity-sau-main-dev-coordinator
2026-01-02 08:21:52.907 P00   INFO: start command end: completed successfully (5ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-02 08:21:52.972 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=1751856-3339ed15 --log-level-console=info --log-level-file=debug --no-online --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-02 08:21:52.973 P00   INFO: stanza-upgrade for stanza 'identity-sau-main-dev-coordinator' on repo1
2026-01-02 08:21:52.997 P00   INFO: stanza-upgrade command end: completed successfully (30ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-02 08:21:53 UTC] USER=www-data EUID=0 PID=1751860 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260102-082153.log
[2026-01-02 08:21:53 UTC] USER=www-data EUID=0 PID=1751871 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260102-082153.log
[2026-01-02 08:21:53 UTC] USER=www-data EUID=0 PID=1751880 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260102-082153.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-01-02 08:22:04 UTC] USER=www-data EUID=0 PID=1751996 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-1751173.log /var/log/pgbackrest/initial-backup-20260102-082153.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260102-082153.log
   2026-01-02 08:22:04.741 P00   INFO: repo1: remove expired backup 20251205-082040F
   2026-01-02 08:22:04.794 P00   INFO: repo1: 17-22 remove archive, start = 000000010000000000000003, stop = 000000010000000000000005
   2026-01-02 08:22:04.796 P00   INFO: repo1: 17-23 no archive to remove
   2026-01-02 08:22:04.796 P00   INFO: repo1: 17-24 remove archive, start = 000000010000000000000003, stop = 000000010000000000000003
   2026-01-02 08:22:04.796 P00   INFO: expire command end: completed successfully (71ms)

[INFO] Current backups:
stanza: identity-sau-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000006/00000001000000000000000E

        full backup: 20251205-082103F
            timestamp start/stop: 2025-12-05 08:21:03+00 / 2025-12-05 08:21:06+00
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 33.6MB, database backup size: 33.6MB
            repo1: backup set size: 5.5MB, backup size: 5.5MB

    db (prior)
        wal archive min/max (17): 000000010000000000000003/000000010000000000000010

        full backup: 20251205-100802F
            timestamp start/stop: 2025-12-05 10:08:02+00 / 2025-12-05 10:08:13+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 33.6MB, database backup size: 33.6MB
            repo1: backup set size: 5.4MB, backup size: 5.4MB

        full backup: 20251205-100826F
            timestamp start/stop: 2025-12-05 10:08:26+00 / 2025-12-05 10:08:29+00
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 33.6MB, database backup size: 33.6MB
            repo1: backup set size: 5.4MB, backup size: 5.4MB

    db (current)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000004

        full backup: 20260102-082153F
            timestamp start/stop: 2026-01-02 08:21:53+00 / 2026-01-02 08:22:04+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         identity-sau-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/identity-sau-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up worker backups for 1 worker(s)...
[INFO] Setting up backup for: worker-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for identity-sau-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-01-02 08:22:05 UTC] USER=www-data EUID=0 PID=1752053 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-02 08:22:05 UTC] USER=www-data EUID=0 PID=1752062 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/identity-sau-main-dev
[2026-01-02 08:22:05 UTC] USER=www-data EUID=0 PID=1752071 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-01-02 08:22:05 UTC] USER=www-data EUID=0 PID=1752080 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-02 08:22:05 UTC] USER=www-data EUID=0 PID=1752089 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-02 08:22:05 UTC] USER=www-data EUID=0 PID=1752098 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-02 08:22:14 UTC] USER=www-data EUID=0 PID=1752241 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-02 08:22:14 UTC] USER=www-data EUID=0 PID=1752250 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-02 08:22:14 UTC] USER=www-data EUID=0 PID=1752259 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-01-02 08:22:14 UTC] USER=www-data EUID=0 PID=1752268 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-02 08:22:14 UTC] USER=www-data EUID=0 PID=1752277 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-identity-sau-main-dev
[2026-01-02 08:22:15 UTC] USER=www-data EUID=0 PID=1752300 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-02 08:22:15 UTC] USER=www-data EUID=0 PID=1752309 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-01-02 08:22:15 UTC] USER=www-data EUID=0 PID=1752318 ACTION=fsop ARGS=find /var/lib/postgresql/17/identity-sau-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-01-02 08:22:15 UTC] USER=www-data EUID=0 PID=1752327 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-02 08:22:15 UTC] USER=www-data EUID=0 PID=1752337 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-01-02 08:22:15 UTC] USER=www-data EUID=0 PID=1752346 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-01-02 08:22:15 UTC] USER=www-data EUID=0 PID=1752355 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-02 08:22:15 UTC] USER=www-data EUID=0 PID=1752364 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/PG_VERSION
[2026-01-02 08:22:15 UTC] USER=www-data EUID=0 PID=1752374 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza identity-sau-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-02 08:22:16 UTC] USER=www-data EUID=0 PID=1752442 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-02 08:22:19 UTC] USER=www-data EUID=0 PID=1752464 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 08:22:23 UTC] USER=www-data EUID=0 PID=1752512 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-02 08:22:23 UTC] USER=www-data EUID=0 PID=1752537 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator --log-level-console=info check
2026-01-02 08:22:23.436 P00   INFO: check command begin 2.56.0: --exec-id=1752545-fbd825f8 --log-level-console=info --log-level-file=debug --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-02 08:22:23.484 P00   INFO: check repo1 configuration (primary)
2026-01-02 08:22:23.553 P00   INFO: check repo1 archive for WAL (primary)
2026-01-02 08:22:24.155 P00   INFO: WAL segment 000000010000000000000006 successfully archived to '/var/lib/pgbackrest/backup/identity-sau-main-dev/archive/identity-sau-main-dev-coordinator/17-24/0000000100000000/000000010000000000000006-d7012b825614cd75e7a88aa29841208a38d29f4a.lz4' on repo1
2026-01-02 08:22:24.155 P00   INFO: check command end: completed successfully (725ms)
[INFO] ✅ Stanza verification passed

[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-02 08:22:24 UTC] USER=www-data EUID=0 PID=1752576 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-02 08:22:24 UTC] USER=www-data EUID=0 PID=1752585 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-02 08:22:24 UTC] USER=www-data EUID=0 PID=1752603 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[2026-01-02 08:22:24 UTC] USER=www-data EUID=0 PID=1752612 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-02 08:22:24 UTC] USER=www-data EUID=0 PID=1752630 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-identity-sau-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-01-02 08:22:24 UTC] USER=www-data EUID=0 PID=1752648 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 08:22:24 UTC] USER=www-data EUID=0 PID=1752659 ACTION=fsop ARGS=sed -i s|__ENV_ID__|identity-sau-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 08:22:24 UTC] USER=www-data EUID=0 PID=1752668 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/var/lib/postgresql/17/identity-sau-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 08:22:24 UTC] USER=www-data EUID=0 PID=1752677 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 08:22:24 UTC] USER=www-data EUID=0 PID=1752686 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-02 08:22:24.829 P00   INFO: start command begin 2.56.0: --exec-id=1752707-b27d994c --log-level-console=info --log-level-file=debug --stanza=identity-sau-main-dev-coordinator
2026-01-02 08:22:24.830 P00   WARN: stop file does not exist for stanza identity-sau-main-dev-coordinator
2026-01-02 08:22:24.830 P00   INFO: start command end: completed successfully (6ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-02 08:22:24.898 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=1752718-da4f2b87 --log-level-console=info --log-level-file=debug --no-online --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-02 08:22:24.900 P00   INFO: stanza-upgrade for stanza 'identity-sau-main-dev-coordinator' on repo1
2026-01-02 08:22:24.902 P00   INFO: stanza 'identity-sau-main-dev-coordinator' on repo1 is already up to date
2026-01-02 08:22:24.902 P00   INFO: stanza-upgrade command end: completed successfully (9ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-02 08:22:24 UTC] USER=www-data EUID=0 PID=1752722 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260102-082224.log
[2026-01-02 08:22:24 UTC] USER=www-data EUID=0 PID=1752731 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260102-082224.log
[2026-01-02 08:22:25 UTC] USER=www-data EUID=0 PID=1752740 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260102-082224.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-01-02 08:22:32 UTC] USER=www-data EUID=0 PID=1752906 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-1752022.log /var/log/pgbackrest/initial-backup-20260102-082224.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260102-082224.log
   2026-01-02 08:22:32.766 P00   INFO: repo1: remove expired backup 20251205-082103F
   2026-01-02 08:22:32.839 P00   INFO: repo1: remove archive path /var/lib/pgbackrest/backup/identity-sau-main-dev/archive/identity-sau-main-dev-coordinator/17-22
   2026-01-02 08:22:32.846 P00   INFO: repo1: 17-23 no archive to remove
   2026-01-02 08:22:32.846 P00   INFO: repo1: 17-24 no archive to remove
   2026-01-02 08:22:32.847 P00   INFO: expire command end: completed successfully (129ms)

[INFO] Current backups:
stanza: identity-sau-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000003/000000010000000000000010

        full backup: 20251205-100802F
            timestamp start/stop: 2025-12-05 10:08:02+00 / 2025-12-05 10:08:13+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 33.6MB, database backup size: 33.6MB
            repo1: backup set size: 5.4MB, backup size: 5.4MB

        full backup: 20251205-100826F
            timestamp start/stop: 2025-12-05 10:08:26+00 / 2025-12-05 10:08:29+00
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 33.6MB, database backup size: 33.6MB
            repo1: backup set size: 5.4MB, backup size: 5.4MB

    db (current)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000007

        full backup: 20260102-082153F
            timestamp start/stop: 2026-01-02 08:21:53+00 / 2026-01-02 08:22:04+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

        full backup: 20260102-082225F
            timestamp start/stop: 2026-01-02 08:22:25+00 / 2026-01-02 08:22:32+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         identity-sau-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/identity-sau-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ Backup setup completed for coordinator and all workers

[INFO] Skipping 06-distribute-tables-canary.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 07-distribute-tables.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:22:34 UTC] USER=unknown EUID=33 PID=1752969 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 08:22:34 UTC] USER=unknown EUID=33 PID=1752976 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 08:22:34 UTC] USER=unknown EUID=33 PID=1752983 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 08:22:34 UTC] USER=unknown EUID=33 PID=1752990 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS TABLE DISTRIBUTION
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔐 Secure connection established
[INFO]    Host: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432
[INFO]    Database: fastorder_identity_sau_main_dev_db
[INFO]    SSL: verify-full (TLS 1.2+)
[INFO]    Timeouts: statement=120s, idle_tx=300s

[INFO] 🔍 Running preflight checks...
[INFO] Testing database connectivity...
[OK]   ✅ Database connection successful
[OK]   ✅ Connected to correct database: fastorder_identity_sau_main_dev_db
[INFO] Checking Citus extension in database fastorder_identity_sau_main_dev_db...
[OK]   Citus version: 13.2-1
[INFO] Checking worker registration...
[OK]   Registered workers: 1
[INFO] Worker nodes:
[INFO]                             nodename                           | nodeport | isactive | noderole 
[INFO]   -------------------------------------------------------------+----------+----------+----------
[INFO]    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com |     5432 | t        | primary
[INFO]   (1 row)
[INFO]   

[INFO] 📊 Starting table distribution...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Distributing: auth.login_account
[INFO] Description: User authentication table - distributed by region for tenant isolation
[INFO] Shard key: region_hint
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📝 Current rows: 0
[INFO] Checking constraints compatibility with Citus...
[OK]   ✅ No conflicting constraints found
[OK]   ✅ Table already distributed - skipping
[INFO]    Distribution column: region_hint
[OK]   ✅ Data integrity verified (0 rows)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[OK]   ✅ All tables distributed successfully!
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 📊 Citus Cluster Summary:

[INFO] Distributed tables:
[INFO]            table          |   type    | shard_key | shards | size  
[INFO]   ------------------------+-----------+-----------+--------+-------
[INFO]    core.tenant            | reference | <none>    |      1 | 24 kB
[INFO]    core.realm             | local     | <none>    |      1 | 40 kB
[INFO]    core.identity          | local     | <none>    |      1 | 72 kB
[INFO]    core.device            | local     | <none>    |      1 | 48 kB
[INFO]    core.identity_account  | local     | <none>    |      1 | 48 kB
[INFO]    core.identity_mfa      | local     | <none>    |      1 | 40 kB
[INFO]    core.external_idp_link | local     | <none>    |      1 | 48 kB
[INFO]    policy.client          | local     | <none>    |      1 | 56 kB
[INFO]    policy.resource        | local     | <none>    |      1 | 48 kB
[INFO]    policy.scope           | local     | <none>    |      1 | 40 kB
[INFO]    policy.permission      | local     | <none>    |      1 | 48 kB
[INFO]    policy.role            | local     | <none>    |      1 | 56 kB
[INFO]    policy.role_permission | local     | <none>    |      1 | 24 kB
[INFO]    policy.identity_role   | local     | <none>    |      1 | 40 kB
[INFO]    policy.policy_rule     | local     | <none>    |      1 | 48 kB
[INFO]    policy.api_key         | local     | <none>    |      1 | 56 kB
[INFO]    auth.login_account     | reference | <none>    |      1 | 48 kB
[INFO]   (17 rows)
[INFO]   

[INFO] Worker capacity:
[INFO]    worker | total_shards | total_size 
[INFO]   --------+--------------+------------
[INFO]   (0 rows)
[INFO]   

[OK]   Citus table distribution complete

[INFO] Skipping 08-distribute-tables-rollback.sh (rollback script - run manually only)
[INFO] Skipping 09-distribute-tables-test.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 10-setup-cdc.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] CDC PIPELINE SETUP (Debezium + Elasticsearch Sink)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Log file: /var/log/fastorder/cdc/10-setup-cdc-*.log

[INFO] Running CDC setup for identifier: coordinator
[2026-01-02 08:22:47] ==========================================
[2026-01-02 08:22:47] CDC SETUP SCRIPT STARTED
[2026-01-02 08:22:47] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260102_082247.log
[2026-01-02 08:22:47] ==========================================
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 08:22:47] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:22:47]   CDC Pipeline Setup (Debezium + ES Sink)
[2026-01-02 08:22:47] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:22:47]   Environment: identity-sau-main-dev
[2026-01-02 08:22:47]   Identifier:  coordinator
[2026-01-02 08:22:47]   Service:     identity
[2026-01-02 08:22:47] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:22:47] 📂 CDC_BASE_DIR exists: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc
[2026-01-02 08:22:47] Looking for service folder: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity
[2026-01-02 08:22:47] 
[2026-01-02 08:22:47] 📂 Found CDC configuration for service: identity
[2026-01-02 08:22:47] Scanning for subservice directories in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity
[2026-01-02 08:22:47] Found subservice: login, checking for steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps
[2026-01-02 08:22:47] 
[2026-01-02 08:22:47] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:22:47]   Setting up CDC for: identity/login
[2026-01-02 08:22:47] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:22:47] Found 7 step script(s) in /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps
[2026-01-02 08:22:47] 
[2026-01-02 08:22:47] 🔧 Running: 01-setup-debezium-auth-login.sh
[2026-01-02 08:22:47]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/01-setup-debezium-auth-login.sh
[2026-01-02 08:22:47]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Debezium CDC Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔍 Verifying Kafka infrastructure...
✅ db-identity-sau-main-dev-postgresql.fastorder.com resolves to 10.100.1.213
🔐 psql will use client cert for mTLS.
🔐 Retrieving credentials from secrets vault...
   Clearing cached credentials for coordinator...
✅ Credentials retrieved from secrets vault
🔐 Syncing debezium_user password in PostgreSQL...
✅ debezium_user password synchronized
🔍 Checking PostgreSQL SSL status...
✅ Server SSL is ON (verify-full + client cert).
🔧 Applying publication & grants over TLS…
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

NOTICE:  publication "cdc_pub_identity" does not exist, skipping
DROP PUBLICATION
CREATE PUBLICATION
SET
NOTICE:  Added shard table auth.login_account_102024 to publication
DO
RESET
GRANT
GRANT
GRANT
✅ Publication & grants done (including Citus shard table).
⏳ Waiting for Kafka Connect @ https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors…
[2026-01-02 08:22:53] 🔗 Waiting for Kafka Connect at: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
[2026-01-02 08:22:53] ⏳ Waiting for HTTP endpoint: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
[2026-01-02 08:22:53]    Expected codes: 200,500, timeout: 300s
[2026-01-02 08:22:53] ✅ HTTP endpoint ready: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083 (code: 200, took: 0s)
[2026-01-02 08:22:53] 🔄 Testing Connect worker readiness...
[2026-01-02 08:22:53] ✅ Kafka Connect worker ready
🧹 Cleaning up existing Debezium connector and slot (if any)...
   Step 0a: Also resetting ES Sink connector offsets (required for coordinated reset)...
   → Stopping ES Sink connector pg_identity_sau_main_dev_coordinator_es_sink...
   → Deleting ES Sink connector offsets...
   ✓ ES Sink offsets deleted successfully (HTTP 200)
   → Deleting ES Sink connector (will be recreated by 02-setup-es-sink.sh)...
   ✓ ES Sink connector cleanup complete
   Step 0b: Clearing stale Debezium connector offsets from Kafka Connect...
   → Stopping connector pg_identity_sau_main_dev_debezium_postgres...
   → Deleting connector offsets (forces fresh snapshot)...
   ✓ Connector offsets deleted successfully (HTTP 200)
   Step 1: Ensuring connector is completely removed...
   Deleting connector: pg_identity_sau_main_dev_debezium_postgres (attempt 1/10)
   ✓ Connector pg_identity_sau_main_dev_debezium_postgres does not exist (HTTP 404)
   Step 2: Waiting for replication slot to become inactive...
   ✓ Slot slot_identity_sau_main_dev does not exist (clean state)
   Step 3: Dropping replication slot...
   ✓ Slot slot_identity_sau_main_dev already dropped
   Step 4: Final verification...
✅ Cleanup complete - environment is clean for fresh CDC snapshot
🔐 Checking Debezium SSL certificate permissions...
🔍 Validating Debezium SSL certificates...
🔐 Connector will use mTLS to Postgres.
  ✓ Certificate: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt
  ✓ Key: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key
  ✓ Root CA: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
ℹ️  Skipping pre-flight connectivity test (will be validated by Kafka Connect)
📤 Upserting connector: PUT https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_identity_sau_main_dev_debezium_postgres/config
   Attempt 1/5: Sending PUT request to Kafka Connect...
   (This may take up to 60s as Connect validates the configuration)
   ✅ Success (HTTP 201)

🌐 HTTP Response: 201
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Response body:
{
  "name": "pg_identity_sau_main_dev_debezium_postgres",
  "config": {
    "name": "pg_identity_sau_main_dev_debezium_postgres",
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "plugin.name": "pgoutput",
    "database.hostname": "db-identity-sau-main-dev-postgresql.fastorder.com",
    "database.port": "5432",
    "database.dbname": "fastorder_identity_sau_main_dev_db",
    "database.user": "debezium_user",
    "database.password": "2qsnTDG2gYMLA6qqSOqSN9ZeI",
    "database.sslmode": "verify-full",
    "database.sslrootcert": "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt",
    "database.sslcert": "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt",
    "database.sslkey": "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key",
    "publication.name": "cdc_pub_identity",
    "publication.autocreate.mode": "disabled",
    "slot.name": "slot_identity_sau_main_dev",
    "topic.prefix": "identity_sau_main_dev_cdc",
    "schema.include.list": "auth",
    "table.include.list": "auth.login_account,auth.login_account_[0-9]+",
    "transforms": "unwrap,route",
    "transforms.unwrap.add.fields": "op,ts_ms",
    "transforms.unwrap.delete.handling.mode": "rewrite",
    "transforms.unwrap.drop.tombstones": "false",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
    "transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.route.regex": "^identity_sau_main_dev_cdc\\.auth\\.login_account(_[0-9]+)?$",
    "transforms.route.replacement": "identity_sau_main_dev_account_router",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",
    "snapshot.mode": "always"
  },
  "tasks": [],
  "type": "source"
}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Connector upserted.
🔄 Verifying connector task startup...
✅ Debezium connector task is RUNNING
ℹ️  Source table auth.login_account has 0 rows.
ℹ️  Snapshot will be metadata-only; offsets may stay empty until first change.
⏳ Waiting for Debezium initial snapshot to complete...
   📊 Slot status: restart_lsn=0/8012450, confirmed_flush_lsn=0/8012488
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (0s elapsed)
   ⏳ Snapshot in progress... (5s elapsed)
   ⏳ Snapshot in progress... (10s elapsed)
   📊 Slot status: restart_lsn=0/8012450, confirmed_flush_lsn=0/8012488
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (15s elapsed)
   ⏳ Snapshot in progress... (20s elapsed)
   ⏳ Snapshot in progress... (25s elapsed)
   📊 Slot status: restart_lsn=0/8012450, confirmed_flush_lsn=0/8012488
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (30s elapsed)
   ⏳ Snapshot in progress... (35s elapsed)
   ⏳ Snapshot in progress... (40s elapsed)
   📊 Slot status: restart_lsn=0/8012450, confirmed_flush_lsn=0/8012488
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (45s elapsed)
   ⏳ Snapshot in progress... (50s elapsed)
   ⏳ Snapshot in progress... (55s elapsed)
   📊 Slot status: restart_lsn=0/8012450, confirmed_flush_lsn=0/8012488
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (60s elapsed)
   ⏳ Snapshot in progress... (65s elapsed)
   ⏳ Snapshot in progress... (70s elapsed)
   📊 Slot status: restart_lsn=0/8012450, confirmed_flush_lsn=0/8012488
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (75s elapsed)
   ⏳ Snapshot in progress... (80s elapsed)
   ⏳ Snapshot in progress... (85s elapsed)
   📊 Slot status: restart_lsn=0/8012450, confirmed_flush_lsn=0/8012488
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (90s elapsed)
   ⏳ Snapshot in progress... (95s elapsed)
   ⏳ Snapshot in progress... (100s elapsed)
   📊 Slot status: restart_lsn=0/8012450, confirmed_flush_lsn=0/8012488
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (105s elapsed)
   ⏳ Snapshot in progress... (110s elapsed)
   ⏳ Snapshot in progress... (115s elapsed)

⚠️  WARNING: Snapshot wait timeout (120s) on EMPTY table.
   Offsets are still empty, but source table has 0 rows.
   Proceeding anyway – CDC health will be verified by test inserts.

✅ Debezium connector is RUNNING after snapshot
🔍 Final verification: Checking Debezium offsets are recorded...
   ℹ️  Source table auth.login_account has 0 rows
   ℹ️  Skipping offset verification (no data to snapshot)
✅ Debezium connector verified RUNNING (empty source table)
🔄 Phase 2: Updating connector to snapshot.mode=initial...
✅ Connector updated to snapshot.mode=initial (HTTP 200)
✅ Connector verified RUNNING after Phase 2 update
✅ Debezium connector configured successfully (two-phase snapshot complete)
[2026-01-02 08:25:26] ✅ Completed: 01-setup-debezium-auth-login.sh
[2026-01-02 08:25:26] 
[2026-01-02 08:25:26] 🔧 Running: 02-setup-es-sink.sh
[2026-01-02 08:25:26]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/02-setup-es-sink.sh
[2026-01-02 08:25:26]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[WARN] Master/coordinator not found, using node-01
[INFO] Using ES domain: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com
🔐 Retrieving keystore passwords from secrets manager...
[INFO] Retrieving Kafka truststore password...
✅ Retrieved passwords from remote backend
✅ Retrieved Kafka truststore password
[INFO] Retrieving Elasticsearch P12 password...
[INFO] 🔍 Checking secrets backend (provider: aws)...
✅ Retrieved passwords from remote backend
[INFO] ✅ Using existing passwords from backend
✅ Retrieved/generated Elasticsearch P12 password
✅ Keystore passwords retrieved successfully
   - Kafka truststore password: yOb0eqkA... (32 chars)
   - ES P12 password: 8siDJx7z... (32 chars)
[INFO] 🔐 Clearing cached ES credentials to ensure fresh retrieval...
[INFO] [INFO] ✅ Using ES password from centralized secrets vault (identifier: node-01)
[INFO] 🔐 Verifying Elasticsearch accepts client certificate...
[INFO] ✅ Elasticsearch accepting client certificate
[INFO] 🔐 Setting up ES client keystore using Kafka client certificate...
[INFO]    Certificate: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem (signed by Fastorder RA Root CA)
[INFO] 📋 Creating ES client P12 keystore from Kafka client certificate...
[2026-01-02 08:25:35 UTC] USER=www-data EUID=0 PID=1756113 ACTION=fsop ARGS=mv /tmp/es-client-1755814.p12 /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[2026-01-02 08:25:35 UTC] USER=www-data EUID=0 PID=1756122 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[2026-01-02 08:25:35 UTC] USER=www-data EUID=0 PID=1756131 ACTION=fsop ARGS=chmod 600 /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[INFO] ✅ Created ES client keystore: /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[INFO]    Using Kafka client cert signed by Fastorder RA Root CA
[INFO] ℹ️  Using Kafka truststore and adding ES CA certificate
[2026-01-02 08:25:35 UTC] USER=www-data EUID=0 PID=1756140 ACTION=fsop ARGS=test -f /opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks
[INFO] 📋 Adding ES CA certificate to truststore...
[2026-01-02 08:25:36 UTC] USER=www-data EUID=0 PID=1756186 ACTION=passthru ARGS=sudo -u kafka keytool -import -alias elasticsearch-ca -file /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt -keystore /opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks -storepass yOb0eqkAqtj8HEWebgA7nf04YlqsLw44 -noprompt
Certificate was added to keystore
[INFO] ✅ ES CA added to truststore
[INFO] [INFO] 🔗 Waiting for Kafka Connect at: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
[INFO] [INFO] ✅ Connect HTTP ready (code 200)
[INFO] [INFO] 🔍 Verifying Debezium connector snapshot status...
[INFO] [INFO] ℹ️  Source table auth.login_account has 0 rows.
[INFO] [INFO]    Skipping Debezium snapshot wait (metadata-only snapshot on empty table).
[INFO] [INFO] 🔌 Cleaning up existing ES Sink connector: pg_identity_sau_main_dev_coordinator_es_sink
[INFO] [INFO]    → Deleting connector...
[INFO] [INFO]    HTTP 404 (404 is fine)
[INFO] [INFO] 🔐 Validating Elasticsearch credentials...
[INFO] [INFO] ✅ ES credentials validated successfully
[INFO] [INFO] 🔧 Creating required Elasticsearch ingest pipelines: identity-embed-pipeline-001
[INFO] [INFO] ✅ Pipeline identity-embed-pipeline-001 created successfully
[INFO] [INFO] 🔧 Ensuring CDC index has no default_pipeline requirement...
[INFO] [INFO] ✅ Removed default_pipeline from index (if any)
[INFO] [INFO] 🔧 Ensuring dynamic mapping is enabled...
[INFO] [INFO] ✅ Dynamic mapping enabled for identity_sau_main_dev_account_router
[DEBUG] ES_TRUSTSTORE=/opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks
[DEBUG] ES_CLIENT_P12=/opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[DEBUG] TRUSTSTORE_PASS=yOb0eqkA...
[DEBUG] P12_PASS=8siDJx7z...
== Outgoing connector config (snippet) ==
2:  "name": "pg_identity_sau_main_dev_coordinator_es_sink",
6:  "connection.url": "https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200",
19:  "index": "identity_sau_main_dev_account_router",
[INFO] ⚠️  Skipping pre-validation - will validate on PUT...
[INFO] [INFO] ✅ Proceeding to PUT
[2026-01-02 08:25:38] [1/3] Upserting connector via PUT https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_identity_sau_main_dev_coordinator_es_sink/config
🌐 HTTP 201
✅ Connector created/updated successfully
{
  "name": "pg_identity_sau_main_dev_coordinator_es_sink",
  "config": {
    "name": "pg_identity_sau_main_dev_coordinator_es_sink",
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "tasks.max": "1",
    "topics": "identity_sau_main_dev_account_router",
    "connection.url": "https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200",
    "elastic.security.protocol": "SSL",
    "elastic.https.ssl.hostname.verification": "true",
    "elastic.https.ssl.truststore.location": "/opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks",
    "elastic.https.ssl.truststore.password": "yOb0eqkAqtj8HEWebgA7nf04YlqsLw44",
    "elastic.https.ssl.truststore.type": "JKS",
    "elastic.https.ssl.keystore.location": "/opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12",
    "elastic.https.ssl.keystore.password": "8siDJx7zdDhhu5iMMZwnhZfTaGFSgCvh",
    "elastic.https.ssl.keystore.type": "PKCS12",
    "elastic.username": "elastic",
    "elastic.password": "T+kMy0e84aGeV204NzYK",
    "connection.username": "elastic",
    "connection.password": "T+kMy0e84aGeV204NzYK",
    "index": "identity_sau_main_dev_account_router",
    "key.ignore": "true",
    "schema.ignore": "true",
    "behavior.on.null.values": "delete",
    "write.method": "upsert",
    "type.name": "_doc",
    "max.in.flight.requests": "1",
    "batch.size": "2000",
    "linger.ms": "100",
    "flush.timeout.ms": "60000",
    "max.retries": "10",
    "retry.backoff.ms": "5000",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false"
  },
  "tasks": [],
  "type": "sink"
}
{
  "pg_identity_sau_main_dev_debezium_postgres": {
    "status": {
      "name": "pg_identity_sau_main_dev_debezium_postgres",
      "connector": {
        "state": "RUNNING",
        "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
      },
      "tasks": [
        {
          "id": 0,
          "state": "RUNNING",
          "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
        }
      ],
      "type": "source"
    }
  },
  "pg_identity_sau_to_universe_main_dev_es_sink": {
    "status": {
      "name": "pg_identity_sau_to_universe_main_dev_es_sink",
      "connector": {
        "state": "RUNNING",
        "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
      },
      "tasks": [
        {
          "id": 0,
          "state": "FAILED",
          "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083",
          "trace": "org.apache.kafka.common.KafkaException: Failed to load SSL keystore /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12 of type PKCS12\n\tat org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:380)\n\tat org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:352)\n\tat org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:302)\n\tat org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:162)\n\tat org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:147)\n\tat org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:100)\n\tat io.confluent.connect.elasticsearch.ConfigCallbackHandler.sslContext(ConfigCallbackHandler.java:262)\n\tat io.confluent.connect.elasticsearch.ConfigCallbackHandler.createConnectionManager(ConfigCallbackHandler.java:172)\n\tat io.confluent.connect.elasticsearch.ConfigCallbackHandler.customizeHttpClient(ConfigCallbackHandler.java:95)\n\tat org.elasticsearch.client.RestClientBuilder.createHttpClient(RestClientBuilder.java:320)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.client.RestClientBuilder.build(RestClientBuilder.java:283)\n\tat io.confluent.connect.elasticsearch.ElasticsearchClient.<init>(ElasticsearchClient.java:144)\n\tat io.confluent.connect.elasticsearch.ElasticsearchSinkTask.start(ElasticsearchSinkTask.java:82)\n\tat io.confluent.connect.elasticsearch.ElasticsearchSinkTask.start(ElasticsearchSinkTask.java:54)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:324)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doStart(WorkerTask.java:176)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:225)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:281)\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:238)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\nCaused by: java.nio.file.NoSuchFileException: /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12\n\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\tat java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)\n\tat java.base/java.nio.file.Files.newByteChannel(Files.java:380)\n\tat java.base/java.nio.file.Files.newByteChannel(Files.java:432)\n\tat java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)\n\tat java.base/java.nio.file.Files.newInputStream(Files.java:160)\n\tat org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:373)\n\t... 24 more\n"
        }
      ],
      "type": "sink"
    }
  },
  "pg_identity_sau_main_dev_coordinator_es_sink": {
    "status": {
      "name": "pg_identity_sau_main_dev_coordinator_es_sink",
      "connector": {
        "state": "RUNNING",
        "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
      },
      "tasks": [],
      "type": "sink"
    }
  }
}
[INFO] [INFO] 🔗 Creating ES alias for application compatibility...
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (0s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (5s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (10s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (15s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (20s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (25s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (30s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (35s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (40s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (45s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (50s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (55s)
[WARN] ⚠️  ES index not created within 60s, skipping alias creation

🔍 Final verification: Checking ES document count...
   PostgreSQL auth.login_account: 0 rows
ℹ️  PostgreSQL table is empty - skipping ES verification
✅ Done.
[2026-01-02 08:26:41] ✅ Completed: 02-setup-es-sink.sh
[2026-01-02 08:26:41] 
[2026-01-02 08:26:41] 🔧 Running: 03-setup-es-universe-sink.sh
[2026-01-02 08:26:41]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/03-setup-es-universe-sink.sh
[2026-01-02 08:26:41]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Universe Identity ES Sink Setup (Dual-Sink Pattern)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Source Zone:  sau
  Connector:      pg_identity_sau_to_universe_main_dev_es_sink
  Source Topic:   identity_sau_main_dev_account_router
  Universe ES:      search-identity-universe-main-dev.fastorder.com:9200
  Universe Index:   identity_universe_main_dev_account_router
  Zone Field:   zone: "sau" (added to each document)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Retrieving keystore passwords from secrets manager...
✅ Retrieved passwords from remote backend
✅ Retrieved Kafka truststore password
[INFO] 🔍 Checking secrets backend (provider: aws)...
✅ Retrieved passwords from remote backend
[INFO] ✅ Using existing passwords from backend
✅ Retrieved/generated Elasticsearch P12 password
🔐 Retrieving Universe ES password...
[INFO] [INFO] ✅ Retrieved Universe ES password from vault (identifier: node-01)
❌ missing CA file: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:26:50] ❌ FAILED: 03-setup-es-universe-sink.sh (exit code: 1)
[2026-01-02 08:26:50] ❌ CRITICAL: This is a required step for CDC pipeline. Aborting.

[ERROR] ❌ Database infrastructure (postgresql) setup failed with exit code: 1