📊 Provisioning Job Status

[INFO] Using database engine from DB_ENGINE environment variable: postgresql
[INFO] Cleaning up any existing locks...

Starting database engine: postgresql
═══════════════════════════════════════════════

[INFO] Loaded from topology.json: identity-sau-main-dev
[2026-01-02 08:38:59] Loaded environment: identity-sau-main-dev
[2026-01-02 08:38:59] Service: identity, Zone: sau, Branch: main, Env: dev
[2026-01-02 08:38:59] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-02 08:38:59] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-02 08:38:59] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[INFO] Observability cell verified for identity-sau-main-dev
[INFO] Monitoring will be configured after PostgreSQL deployment (step 10-monitoring-setup.sh)
[INFO] Citus mode ENABLED
[INFO] → Coordinator + 1 worker(s) + 1 standby node(s) per worker
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up coordinator (Citus control plane)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-02 08:39:00 UTC] USER=unknown EUID=33 PID=1770012 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 08:39:00 UTC] USER=unknown EUID=33 PID=1770022 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 08:39:00 UTC] USER=unknown EUID=33 PID=1770040 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-02 08:39:00 UTC] USER=unknown EUID=33 PID=1770048 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 08:39:00 UTC] USER=unknown EUID=33 PID=1770055 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-02 08:39:00 UTC] USER=unknown EUID=33 PID=1770062 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4 JOB_UUID=ccddb67e-4182-4074-b2a8-add87af57fe4

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] PostgreSQL IP: 10.100.1.213
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

[INFO] Adding /etc/hosts entries for coordinator...
[INFO]   1. db-identity-sau-main-dev-postgresql.fastorder.com → 10.100.1.213 (primary/short)
[INFO]   2. db-identity-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.213 (compatibility)

[INFO]   ✅ db-identity-sau-main-dev-postgresql.fastorder.com already exists with correct IP
[INFO]   ✅ db-identity-sau-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.213    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  10.100.1.213    db-identity-sau-main-dev-postgresql.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 08:39:04 UTC] USER=www-data EUID=0 PID=1770335 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator and /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:39:04 UTC] USER=www-data EUID=0 PID=1770344 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-02 08:39:04 UTC] USER=www-data EUID=0 PID=1770354 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1770301
[2026-01-02 08:39:04 UTC] USER=www-data EUID=0 PID=1770363 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1770301/ra_root.crt
[2026-01-02 08:39:04 UTC] USER=www-data EUID=0 PID=1770372 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1770301/ra_root.key
[2026-01-02 08:39:04 UTC] USER=www-data EUID=0 PID=1770381 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1770301/ra_root.crt
[2026-01-02 08:39:04 UTC] USER=www-data EUID=0 PID=1770391 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1770301/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-02 08:39:05 UTC] USER=www-data EUID=0 PID=1770429 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1770301/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 08:39:05 UTC] USER=www-data EUID=0 PID=1770438 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1770301/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 08:39:06 UTC] USER=www-data EUID=0 PID=1770447 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-02 08:39:06 UTC] USER=www-data EUID=0 PID=1770456 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1770301/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:06 UTC] USER=www-data EUID=0 PID=1770465 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:06 UTC] USER=www-data EUID=0 PID=1770474 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:06 UTC] USER=www-data EUID=0 PID=1770483 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 08:39:06 UTC] USER=www-data EUID=0 PID=1770494 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 08:39:06 UTC] USER=www-data EUID=0 PID=1770503 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 08:39:06 UTC] USER=www-data EUID=0 PID=1770512 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 08:39:06 UTC] USER=www-data EUID=0 PID=1770521 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 08:39:06 UTC] USER=www-data EUID=0 PID=1770530 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:39:06 UTC] USER=www-data EUID=0 PID=1770539 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        coordinator
Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:39:07 UTC] USER=www-data EUID=0 PID=1770597 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-01-02 08:39:07 UTC] USER=www-data EUID=0 PID=1770608 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 08:39:07 UTC] USER=www-data EUID=0 PID=1770617 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-01-02 08:39:07 UTC] USER=www-data EUID=0 PID=1770626 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 08:39:07 UTC] USER=www-data EUID=0 PID=1770635 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:39:07 UTC] USER=www-data EUID=0 PID=1770651 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:39:07 UTC] USER=www-data EUID=0 PID=1770660 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:39:07 UTC] USER=www-data EUID=0 PID=1770669 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:39:07 UTC] USER=www-data EUID=0 PID=1770678 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:39:07 UTC] USER=www-data EUID=0 PID=1770687 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:07 UTC] USER=www-data EUID=0 PID=1770696 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:39:07 UTC] USER=www-data EUID=0 PID=1770705 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:39:07 UTC] USER=www-data EUID=0 PID=1770714 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:39:08 UTC] USER=www-data EUID=0 PID=1770725 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:39:08 UTC] USER=www-data EUID=0 PID=1770734 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:08 UTC] USER=www-data EUID=0 PID=1770743 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:39:08 UTC] USER=www-data EUID=0 PID=1770752 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:39:08 UTC] USER=www-data EUID=0 PID=1770761 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:39:08 UTC] USER=www-data EUID=0 PID=1770770 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:39:08 UTC] USER=www-data EUID=0 PID=1770779 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:08 UTC] USER=www-data EUID=0 PID=1770788 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:08 UTC] USER=www-data EUID=0 PID=1770814 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:08 UTC] USER=www-data EUID=0 PID=1770823 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:39:08 UTC] USER=www-data EUID=0 PID=1770833 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:39:08 UTC] USER=www-data EUID=0 PID=1770842 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:08 UTC] USER=www-data EUID=0 PID=1770851 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:08 UTC] USER=www-data EUID=0 PID=1770860 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:39:08 UTC] USER=www-data EUID=0 PID=1770871 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:39:08 UTC] USER=www-data EUID=0 PID=1770880 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:08 UTC] USER=www-data EUID=0 PID=1770889 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1770898 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1770907 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1770917 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1770927 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1770936 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1770945 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1770956 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1770965 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1770974 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1770983 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1770992 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1771001 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1771010 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1771019 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1771029 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1771039 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1771049 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1771058 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1771067 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1771076 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1771085 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:39:09 UTC] USER=www-data EUID=0 PID=1771094 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771105 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771114 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771123 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771132 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771142 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771152 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771162 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771171 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771180 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771189 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771198 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771207 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771216 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771225 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771234 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771243 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771253 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771263 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771272 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771281 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771290 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771299 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771311 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771320 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771329 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:39:10 UTC] USER=www-data EUID=0 PID=1771338 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:39:11 UTC] USER=www-data EUID=0 PID=1771383 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-01-02 08:39:11 UTC] USER=www-data EUID=0 PID=1771392 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 08:39:11 UTC] USER=www-data EUID=0 PID=1771401 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-01-02 08:39:11 UTC] USER=www-data EUID=0 PID=1771410 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 08:39:11 UTC] USER=www-data EUID=0 PID=1771419 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771435 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771444 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771453 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771462 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771472 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771482 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771491 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771500 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771509 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771518 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771529 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771538 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771547 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771557 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771566 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771575 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771584 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771593 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771619 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771628 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771637 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771646 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771655 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:12 UTC] USER=www-data EUID=0 PID=1771664 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771675 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771684 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771693 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771702 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771711 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771721 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771731 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771740 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771749 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771758 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771767 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771776 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771785 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771794 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771803 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771812 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771821 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771831 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771841 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771850 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771859 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771868 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771879 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771888 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771897 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:39:13 UTC] USER=www-data EUID=0 PID=1771906 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1771915 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1771925 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1771934 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1771944 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1771954 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1771963 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1771972 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1771981 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1771990 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1772001 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1772010 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1772019 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1772028 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1772037 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1772046 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1772056 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1772066 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1772075 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1772084 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1772093 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1772102 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1772111 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1772120 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1772129 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:39:14 UTC] USER=www-data EUID=0 PID=1772139 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-coordinator-postgresql environment: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com (10.100.1.213)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.213
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] Data dir:   /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[INFO] Port:       5432
[INFO] Hostname:   db-identity-sau-main-dev-postgresql-coordinator
[2026-01-02 08:39:16 UTC] USER=www-data EUID=0 PID=1772238 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:39:16 UTC] USER=www-data EUID=0 PID=1772259 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:39:17 UTC] USER=www-data EUID=0 PID=1772280 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:39:17 UTC] USER=www-data EUID=0 PID=1772301 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 08:39:17 UTC] USER=www-data EUID=0 PID=1772343 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator and /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:39:17 UTC] USER=www-data EUID=0 PID=1772352 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-02 08:39:17 UTC] USER=www-data EUID=0 PID=1772362 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1772308
[2026-01-02 08:39:17 UTC] USER=www-data EUID=0 PID=1772371 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1772308/ra_root.crt
[2026-01-02 08:39:17 UTC] USER=www-data EUID=0 PID=1772382 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1772308/ra_root.key
[2026-01-02 08:39:17 UTC] USER=www-data EUID=0 PID=1772391 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1772308/ra_root.crt
[2026-01-02 08:39:17 UTC] USER=www-data EUID=0 PID=1772400 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1772308/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-02 08:39:19 UTC] USER=www-data EUID=0 PID=1772443 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1772308/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 08:39:19 UTC] USER=www-data EUID=0 PID=1772452 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1772308/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 08:39:19 UTC] USER=www-data EUID=0 PID=1772461 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-02 08:39:20 UTC] USER=www-data EUID=0 PID=1772470 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1772308/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:20 UTC] USER=www-data EUID=0 PID=1772479 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:20 UTC] USER=www-data EUID=0 PID=1772488 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:39:20 UTC] USER=www-data EUID=0 PID=1772497 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 08:39:20 UTC] USER=www-data EUID=0 PID=1772508 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 08:39:20 UTC] USER=www-data EUID=0 PID=1772517 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 08:39:20 UTC] USER=www-data EUID=0 PID=1772526 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 08:39:20 UTC] USER=www-data EUID=0 PID=1772535 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 08:39:20 UTC] USER=www-data EUID=0 PID=1772546 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:39:20 UTC] USER=www-data EUID=0 PID=1772555 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        coordinator
Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-02 08:39:20 UTC] USER=www-data EUID=0 PID=1772584 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 08:39:20 UTC] USER=www-data EUID=0 PID=1772593 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 08:39:20 UTC] USER=www-data EUID=0 PID=1772602 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-02 08:39:20 UTC] USER=www-data EUID=0 PID=1772623 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 08:39:20 UTC] USER=www-data EUID=0 PID=1772648 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-02 08:39:21 UTC] USER=www-data EUID=0 PID=1772679 ACTION=fsop ARGS=rm -rf /var/run/postgresql-identity-sau-main-dev-coordinator
[OK]   No conflicting Postgres left on port 5432
[OK]   Using postgres password from vault provider
[2026-01-02 08:39:24 UTC] USER=www-data EUID=0 PID=1772847 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.nc4PQz
[2026-01-02 08:39:24 UTC] USER=www-data EUID=0 PID=1772868 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.nc4PQz
[2026-01-02 08:39:24 UTC] USER=www-data EUID=0 PID=1772890 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 08:39:24 UTC] USER=www-data EUID=0 PID=1772912 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 08:39:24 UTC] USER=www-data EUID=0 PID=1772934 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/identity-sau-main-dev
[INFO] Initializing cluster in /var/lib/postgresql/17/identity-sau-main-dev/coordinator (SCRAM; pwfile)
[WARN] Removing existing data directory: /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 08:39:24 UTC] USER=www-data EUID=0 PID=1772955 ACTION=fsop ARGS=rm -rf /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 08:39:25 UTC] USER=www-data EUID=0 PID=1772977 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 08:39:25 UTC] USER=www-data EUID=0 PID=1772998 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 08:39:25 UTC] USER=www-data EUID=0 PID=1773020 ACTION=fsop ARGS=chmod 700 /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 08:39:25 UTC] USER=www-data EUID=0 PID=1773044 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-02 08:39:25 UTC] USER=www-data EUID=0 PID=1773066 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-02 08:39:25 UTC] USER=www-data EUID=0 PID=1773088 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-02 08:39:25 UTC] USER=www-data EUID=0 PID=1773097 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /var/lib/postgresql/17/identity-sau-main-dev/coordinator --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.nc4PQz
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/17/identity-sau-main-dev/coordinator ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /var/lib/postgresql/17/identity-sau-main-dev/coordinator -l logfile start

[OK]   initdb complete
[2026-01-02 08:39:26 UTC] USER=www-data EUID=0 PID=1773138 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.nc4PQz
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-02 08:39:27 UTC] USER=www-data EUID=0 PID=1773240 ACTION=fsop ARGS=cp /tmp/tmp.hE2mV8jfLY /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
[2026-01-02 08:39:27 UTC] USER=www-data EUID=0 PID=1773306 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
[2026-01-02 08:39:27 UTC] USER=www-data EUID=0 PID=1773328 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 08:39:27 UTC] USER=www-data EUID=0 PID=1773353 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.Cam7Oe /etc/systemd/system/postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 08:39:27 UTC] USER=www-data EUID=0 PID=1773376 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@identity-sau-main-dev-coordinator.service
[OK]   systemd unit written
[2026-01-02 08:39:27 UTC] USER=www-data EUID=0 PID=1773398 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 08:39:27 UTC] USER=www-data EUID=0 PID=1773420 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 08:39:27 UTC] USER=www-data EUID=0 PID=1773441 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-02 08:39:29 UTC] USER=www-data EUID=0 PID=1773563 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-02 08:39:30 UTC] USER=www-data EUID=0 PID=1773609 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Provisioning application database and Debezium role (if not exists)...
[INFO] Checking if database fastorder_identity_sau_main_dev_db exists...
[INFO] DB check result: exit_code=0, output='[2026-01-02 08:39:31 UTC] USER=www-data EUID=0 PID=1773870 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_identity_sau_main_dev_db''
[INFO] Creating database fastorder_identity_sau_main_dev_db...
[2026-01-02 08:39:31 UTC] USER=www-data EUID=0 PID=1773895 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_identity_sau_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[OK]   Database fastorder_identity_sau_main_dev_db created
[INFO] Checking if role debezium_user exists...
[INFO] Role check result: exit_code=0, output='[2026-01-02 08:39:32 UTC] USER=www-data EUID=0 PID=1773921 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[INFO] Creating role debezium_user...
[2026-01-02 08:39:32 UTC] USER=www-data EUID=0 PID=1773949 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'M1LuGEtYUPCJkG5A9b4B+tYC';
CREATE ROLE
[OK]   Role debezium_user created
[2026-01-02 08:39:32 UTC] USER=www-data EUID=0 PID=1773972 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_identity_sau_main_dev_db" TO debezium_user;
GRANT
[OK]   Application DB (fastorder_identity_sau_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[INFO] Applying connection and memory optimizations...
[INFO] Current settings: max_connections=100, work_mem=4MB
[INFO] Target settings (coordinator): max_connections=150, work_mem=8MB
[2026-01-02 08:39:32 UTC] USER=www-data EUID=0 PID=1774052 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 150;
ALTER SYSTEM
[2026-01-02 08:39:33 UTC] USER=www-data EUID=0 PID=1774077 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-02 08:39:33 UTC] USER=www-data EUID=0 PID=1774103 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   Settings applied to postgresql.auto.conf
[2026-01-02 08:39:33 UTC] USER=www-data EUID=0 PID=1774118 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/standby.signal
[INFO] Service recently started (3s ago) - restarting to apply max_connections...
[INFO] Stopping service...
[2026-01-02 08:39:33 UTC] USER=www-data EUID=0 PID=1774140 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[INFO] Waiting for port 5432 to be released...
[OK]   Port 5432 released
[INFO] Starting service...
[2026-01-02 08:39:36 UTC] USER=www-data EUID=0 PID=1774225 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 08:39:42 UTC] USER=www-data EUID=0 PID=1774297 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[OK]   ✅ Optimization complete: max_connections=150, work_mem=8MB
[INFO] Setting postgres password via centralized script... for coordinator
[INFO] Temporarily disabling synchronous_commit on coordinator for password setting...
[OK]   Disabled synchronous_commit (was: on)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Initial setup: Using password from initdb
✓ PostgreSQL password already set during initdb
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/coordinator
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
[INFO] Restoring synchronous_commit on coordinator...
[OK]   Restored synchronous_commit to: on
[OK]   Password set and persisted
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] PostgreSQL IP: 10.100.1.213
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

[INFO] Adding /etc/hosts entries for coordinator...
[INFO]   1. db-identity-sau-main-dev-postgresql.fastorder.com → 10.100.1.213 (primary/short)
[INFO]   2. db-identity-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.213 (compatibility)

[INFO]   ✅ db-identity-sau-main-dev-postgresql.fastorder.com already exists with correct IP
[INFO]   ✅ db-identity-sau-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.213    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  10.100.1.213    db-identity-sau-main-dev-postgresql.fastorder.com


[OK]   PostgreSQL 'identity-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key \
        host=db-identity-sau-main-dev-postgresql-coordinator port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.213
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-coordinator
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: ce097707-5ce5-40c8-a941-01512555cab8
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 📦 03 role...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 08:39:57 UTC] USER=www-data EUID=0 PID=1774791 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    debezium_user
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   debezium_user
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:40:20 UTC] USER=www-data EUID=0 PID=1775082 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-debezium_user
[2026-01-02 08:40:20 UTC] USER=www-data EUID=0 PID=1775091 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-01-02 08:40:20 UTC] USER=www-data EUID=0 PID=1775100 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-debezium_user/ra_root.key
[2026-01-02 08:40:20 UTC] USER=www-data EUID=0 PID=1775118 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:40:21 UTC] USER=www-data EUID=0 PID=1775134 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:40:21 UTC] USER=www-data EUID=0 PID=1775144 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:40:21 UTC] USER=www-data EUID=0 PID=1775153 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 08:40:21 UTC] USER=www-data EUID=0 PID=1775162 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 08:40:21 UTC] USER=www-data EUID=0 PID=1775171 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:40:21 UTC] USER=www-data EUID=0 PID=1775180 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:40:21 UTC] USER=www-data EUID=0 PID=1775189 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 08:40:21 UTC] USER=www-data EUID=0 PID=1775198 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 08:40:21 UTC] USER=www-data EUID=0 PID=1775207 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 08:40:21 UTC] USER=www-data EUID=0 PID=1775216 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:40:21 UTC] USER=www-data EUID=0 PID=1775225 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:40:21 UTC] USER=www-data EUID=0 PID=1775234 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 08:40:21 UTC] USER=www-data EUID=0 PID=1775243 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 08:40:21 UTC] USER=www-data EUID=0 PID=1775252 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 08:40:21 UTC] USER=www-data EUID=0 PID=1775261 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:40:21 UTC] USER=www-data EUID=0 PID=1775271 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775299 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775308 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775317 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775326 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775335 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775344 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775355 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775364 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775373 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775382 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775391 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775403 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775415 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775424 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775433 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775460 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775469 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775478 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775487 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775496 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775505 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 08:40:22 UTC] USER=www-data EUID=0 PID=1775515 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775541 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775559 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775586 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775619 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775646 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775678 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775690 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775699 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775709 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775718 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775728 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775738 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775747 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775756 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775765 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775774 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775783 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775792 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775801 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775810 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775819 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775828 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775838 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775848 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:40:23 UTC] USER=www-data EUID=0 PID=1775857 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:40:24 UTC] USER=www-data EUID=0 PID=1775866 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:40:24 UTC] USER=www-data EUID=0 PID=1775875 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:40:24 UTC] USER=www-data EUID=0 PID=1775884 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:40:24 UTC] USER=www-data EUID=0 PID=1775893 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:40:24 UTC] USER=www-data EUID=0 PID=1775902 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:40:24 UTC] USER=www-data EUID=0 PID=1775911 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:40:24 UTC] USER=www-data EUID=0 PID=1775920 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: debezium_user
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U debezium_user -d postgres

✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: fastorder_admin_gd
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: coordinator/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-identity-sau-main-dev-postgresql.fastorder.com
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    fastorder_admin_gd
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   fastorder_admin_gd
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:40:39 UTC] USER=www-data EUID=0 PID=1776582 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-fastorder_admin_gd
[2026-01-02 08:40:39 UTC] USER=www-data EUID=0 PID=1776591 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-02 08:40:39 UTC] USER=www-data EUID=0 PID=1776600 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
[2026-01-02 08:40:39 UTC] USER=www-data EUID=0 PID=1776610 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-02 08:40:39 UTC] USER=www-data EUID=0 PID=1776621 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:40:39 UTC] USER=www-data EUID=0 PID=1776635 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:40:39 UTC] USER=www-data EUID=0 PID=1776644 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776653 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776662 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776672 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776683 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776692 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776701 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776710 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776719 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776728 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776737 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776746 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776755 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776764 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776773 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776782 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776791 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776817 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776826 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776835 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776844 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776853 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776862 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 08:40:40 UTC] USER=www-data EUID=0 PID=1776873 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 08:40:41 UTC] USER=www-data EUID=0 PID=1776882 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:40:41 UTC] USER=www-data EUID=0 PID=1776892 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:40:41 UTC] USER=www-data EUID=0 PID=1776902 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:40:41 UTC] USER=www-data EUID=0 PID=1776911 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 08:40:41 UTC] USER=www-data EUID=0 PID=1776921 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:41 UTC] USER=www-data EUID=0 PID=1776932 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:41 UTC] USER=www-data EUID=0 PID=1776942 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:40:41 UTC] USER=www-data EUID=0 PID=1776951 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:40:41 UTC] USER=www-data EUID=0 PID=1776960 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:41 UTC] USER=www-data EUID=0 PID=1776969 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:41 UTC] USER=www-data EUID=0 PID=1776978 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 08:40:41 UTC] USER=www-data EUID=0 PID=1776987 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 08:40:41 UTC] USER=www-data EUID=0 PID=1776996 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:40:41 UTC] USER=www-data EUID=0 PID=1777005 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:40:41 UTC] USER=www-data EUID=0 PID=1777014 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:40:41 UTC] USER=www-data EUID=0 PID=1777023 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 08:40:41 UTC] USER=www-data EUID=0 PID=1777033 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777043 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777052 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777061 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777070 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777079 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777089 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777099 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777108 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777117 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777126 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777135 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777146 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777159 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777168 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777177 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777186 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777195 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777204 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777213 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777222 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777231 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777240 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777251 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 08:40:42 UTC] USER=www-data EUID=0 PID=1777261 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:40:43 UTC] USER=www-data EUID=0 PID=1777282 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:40:43 UTC] USER=www-data EUID=0 PID=1777291 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:40:43 UTC] USER=www-data EUID=0 PID=1777300 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:40:43 UTC] USER=www-data EUID=0 PID=1777309 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:40:43 UTC] USER=www-data EUID=0 PID=1777318 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:40:43 UTC] USER=www-data EUID=0 PID=1777327 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:40:43 UTC] USER=www-data EUID=0 PID=1777337 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:40:43 UTC] USER=www-data EUID=0 PID=1777347 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: fastorder_admin_gd
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-identity-sau-main-dev-coordinator:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_identity_sau_main_dev_db already exists, skipping creation
[2026-01-02 08:40:43 UTC] USER=www-data EUID=0 PID=1777406 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-02 08:40:44 UTC] USER=www-data EUID=0 PID=1777440 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

=== Pre-flight Checks ===
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
✓ AWS Secrets Manager accessible

=== Retrieving Credentials from AWS ===
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
/opt/fastorder/bash/infra_core/cache.sh: line 145: /var/cache/secrets/fastorder_db_identity_sau_main_dev_postgresql_coordinator_fastorder_admin_gd.cache.tmp.1777458: Permission denied
✅ Retrieved from secrets manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials loaded for coordinator/fastorder_admin_gd: fastorder_admin_gd@db-identity-sau-main-dev-postgresql.fastorder.com:5432/fastorder_identity_sau_main_dev_db
✓ Credentials retrieved: fastorder_admin_gd@db-identity-sau-main-dev-postgresql.fastorder.com:5432/fastorder_identity_sau_main_dev_db
╔════════════════════════════════════════════╗
║  PostgreSQL Test Suite (AWS Secrets MGR)  ║
╚════════════════════════════════════════════╝

=== PostgreSQL Authentication Test ===
✗ PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-identity-sau-main-dev-postgresql.fastorder.com" (10.100.1.213), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
── fast setup ─────────────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : coordinator
  PG HOST     : db-identity-sau-main-dev-postgresql.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_identity_sau_main_dev_db
  SCHEMA      : auth
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
  DNS → 10.100.1.213
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User debezium_user does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: debezium_user
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: coordinator/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-01-02 08:41:06 UTC] USER=www-data EUID=0 PID=1778027 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
[2026-01-02 08:41:06 UTC] USER=www-data EUID=0 PID=1778054 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d fastorder_identity_sau_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=coordinator IDENTIFIER_PARENT=coordinator
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║                    IAM Database Schema Initialization                       ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] 🟢 Starting IAM schema provisioning...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

[INFO] 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: core
  Core Identity Directory (tenants, realms, identities, devices, MFA)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [1/20]: core/01-tenant
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
🔧 Installing extensions...
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
   Creating reference table: core.tenant
 create_reference_table 
------------------------
 
(1 row)

✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[OK] Table core/01-tenant initialized

[INFO] 🔸 Table [2/20]: core/02-realm
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.realm$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_realm_updated" for relation "core.realm" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[OK] Table core/02-realm initialized

[INFO] 🔸 Table [3/20]: core/03-identity
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_identity_updated" for relation "core.identity" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[OK] Table core/03-identity initialized

[INFO] 🔸 Table [4/20]: core/04-device
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.device$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[OK] Table core/04-device initialized

[INFO] 🔸 Table [5/20]: core/05-identity_account
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity_account$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_identity_account_updated" for relation "core.identity_account" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[OK] Table core/05-identity_account initialized

[INFO] 🔸 Table [6/20]: core/06-identity_mfa
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity_mfa$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[OK] Table core/06-identity_mfa initialized

[INFO] 🔸 Table [7/20]: core/07-external_idp_link
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.external_idp_link$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[OK] Table core/07-external_idp_link initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: policy
  RBAC/ABAC Authorization (clients, roles, permissions, policies)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [8/20]: policy/01-client
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.client$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
NOTICE:  trigger "tr_client_updated" for relation "policy.client" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[OK] Table policy/01-client initialized

[INFO] 🔸 Table [9/20]: policy/02-resource
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.resource$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[OK] Table policy/02-resource initialized

[INFO] 🔸 Table [10/20]: policy/03-scope
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.scope$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_scope_updated" for relation "policy.scope" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[OK] Table policy/03-scope initialized

[INFO] 🔸 Table [11/20]: policy/04-permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.permission$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_permission_updated" for relation "policy.permission" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[OK] Table policy/04-permission initialized

[INFO] 🔸 Table [12/20]: policy/05-role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.role$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_role_updated" for relation "policy.role" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[OK] Table policy/05-role initialized

[INFO] 🔸 Table [13/20]: policy/06-role_permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.role_permission$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[OK] Table policy/06-role_permission initialized

[INFO] 🔸 Table [14/20]: policy/07-identity_role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.identity_role$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[OK] Table policy/07-identity_role initialized

[INFO] 🔸 Table [15/20]: policy/08-policy_rule
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.policy_rule$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_policy_rule_updated" for relation "policy.policy_rule" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[OK] Table policy/08-policy_rule initialized

[INFO] 🔸 Table [16/20]: policy/09-api_key
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.api_key$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[OK] Table policy/09-api_key initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: audit
  Audit & Risk Logging (auth events, admin actions, risk decisions)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [17/20]: audit/01-auth_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[OK] Table audit/01-auth_event initialized

[INFO] 🔸 Table [18/20]: audit/02-admin_action
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[OK] Table audit/02-admin_action initialized

[INFO] 🔸 Table [19/20]: audit/03-risk_decision
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[OK] Table audit/03-risk_decision initialized

[INFO] 🔸 Table [20/20]: audit/04-consent_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[OK] Table audit/04-consent_event initialized


════════════════════════════════════════════════════════════════════════════
[OK] ✅ IAM Schema Initialization Complete!
[OK] All 20 tables initialized successfully

Schemas created:
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

Design highlights:
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

════════════════════════════════════════════════════════════════════════════

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=coordinator IDENTIFIER_PARENT=coordinator
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: identity/run.sh started for SERVICE=identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity
🔍 DEBUG_CHECKPOINT_A3: SERVICE_ROOT exists, discovering table folders
🔍 DEBUG_CHECKPOINT_A4: Found subfolder: auth
🔍 DEBUG_CHECKPOINT_A4b: Checking for nested schema layout in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth
🔍 DEBUG_CHECKPOINT_A4c: Found nested steps dir: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps (display: auth/login)
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: auth/login|/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
[INFO] 📚 Detected grouped table folders under identity/: auth/login

🔍 DEBUG_CHECKPOINT_A7: Current IDENTIFIER=coordinator
🔍 DEBUG_CHECKPOINT_A8_PROCEED: Processing tables on coordinator/main node
🔍 DEBUG_CHECKPOINT_A9: Processing table: auth/login at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
[INFO] 🔸 Table group: auth/login
🔍 DEBUG_CHECKPOINT_A10: About to run numbered steps for table: auth/login
🔍 DEBUG_CHECKPOINT_B1: run_all_numbered_steps_in_dir called for dir=/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps table=auth/login
🔍 DEBUG_CHECKPOINT_B2: Found 1 numbered steps: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B3: About to run step: 01-init-schema.sh
Ab substep 0 compelete start
[DEBUG] Tracking substep start: steps/01-install/steps/identity/auth/login/01-init-schema (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
Ab substep 0 compelete start
[INFO] 📦 01 init schema...
Ab substep 1 compelete start
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing auth.login_account table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating auth schema...
NOTICE:  schema "auth" already exists, skipping
CREATE SCHEMA
✅ Schema created
🔧 Creating account_status ENUM...
DO
✅ ENUM created
🔧 Creating auth.login_account table...
CREATE TABLE
✅ Table created (Citus-compatible with region_hint in all constraints)
🔧 Creating indexes...
CREATE INDEX
CREATE INDEX
✅ Indexes created
🔧 Creating Citus REFERENCE table for CDC compatibility...
 create_reference_table 
------------------------
 
(1 row)

✅ Table created as REFERENCE table (replicated to all nodes)
   CDC via Debezium will work correctly on coordinator
🎉 Schema initialization complete for fastorder_identity_sau_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

📊 Registering environment in monitoring database (obs schema)...
   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
   Resource IP: 142.93.238.16
⚠️  Could not connect to monitoring database, skipping registration
   You can manually register later using:
   /opt/fastorder/bash/scripts/env_app_setup/setup/04-postgresql/steps/register-authN-af-aaaa1-dev.sh

==========================================
✅ Schema initialization complete!
==========================================
Ab substep 1 compelete end
Ab substep 2 compelete start
Ab substep 2 compelete end

🔍 DEBUG_CHECKPOINT_B4: Completed step: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B5: All numbered steps completed for /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A11: Completed numbered steps for table: auth/login
compeleted here

🔍 DEBUG_CHECKPOINT_A12: All tables processed
End of 04-postgresql/steps/01-install/steps/identity/run.sh

✓ ✅ Coordinator setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up 1 worker(s) (Citus data nodes)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up worker: worker-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-02 08:42:44 UTC] USER=unknown EUID=33 PID=1781804 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 08:42:44 UTC] USER=unknown EUID=33 PID=1781811 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 08:42:44 UTC] USER=unknown EUID=33 PID=1781818 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-02 08:42:44 UTC] USER=unknown EUID=33 PID=1781825 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 08:42:44 UTC] USER=unknown EUID=33 PID=1781832 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-02 08:42:44 UTC] USER=unknown EUID=33 PID=1781841 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4 JOB_UUID=ccddb67e-4182-4074-b2a8-add87af57fe4

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.214
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01...
[INFO]   db-identity-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.214

[INFO]   ✅ db-identity-sau-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.214    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 08:42:47 UTC] USER=www-data EUID=0 PID=1781964 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:42:47 UTC] USER=www-data EUID=0 PID=1781973 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-02 08:42:47 UTC] USER=www-data EUID=0 PID=1781983 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1781929
[2026-01-02 08:42:47 UTC] USER=www-data EUID=0 PID=1781992 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1781929/ra_root.crt
[2026-01-02 08:42:47 UTC] USER=www-data EUID=0 PID=1782001 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1781929/ra_root.key
[2026-01-02 08:42:47 UTC] USER=www-data EUID=0 PID=1782010 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1781929/ra_root.crt
[2026-01-02 08:42:47 UTC] USER=www-data EUID=0 PID=1782019 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1781929/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[2026-01-02 08:42:50 UTC] USER=www-data EUID=0 PID=1782065 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1781929/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 08:42:50 UTC] USER=www-data EUID=0 PID=1782074 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1781929/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 08:42:50 UTC] USER=www-data EUID=0 PID=1782083 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 08:42:50 UTC] USER=www-data EUID=0 PID=1782092 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1781929/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:42:50 UTC] USER=www-data EUID=0 PID=1782101 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:42:50 UTC] USER=www-data EUID=0 PID=1782110 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:42:50 UTC] USER=www-data EUID=0 PID=1782119 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 08:42:50 UTC] USER=www-data EUID=0 PID=1782131 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 08:42:50 UTC] USER=www-data EUID=0 PID=1782141 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 08:42:50 UTC] USER=www-data EUID=0 PID=1782150 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 08:42:50 UTC] USER=www-data EUID=0 PID=1782159 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 08:42:50 UTC] USER=www-data EUID=0 PID=1782168 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:42:50 UTC] USER=www-data EUID=0 PID=1782177 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01
Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:42:51 UTC] USER=www-data EUID=0 PID=1782233 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-01-02 08:42:51 UTC] USER=www-data EUID=0 PID=1782242 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 08:42:51 UTC] USER=www-data EUID=0 PID=1782251 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-01-02 08:42:51 UTC] USER=www-data EUID=0 PID=1782260 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 08:42:51 UTC] USER=www-data EUID=0 PID=1782270 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:42:51 UTC] USER=www-data EUID=0 PID=1782284 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:42:51 UTC] USER=www-data EUID=0 PID=1782293 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:42:51 UTC] USER=www-data EUID=0 PID=1782302 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:42:51 UTC] USER=www-data EUID=0 PID=1782311 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782321 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782330 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782339 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782348 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782357 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782366 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782375 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782384 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782393 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782402 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782411 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782420 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782446 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782458 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782467 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782478 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782496 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782505 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782514 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782523 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782532 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782541 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782551 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:52 UTC] USER=www-data EUID=0 PID=1782561 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782582 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782592 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782602 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782611 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782620 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782629 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782638 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782647 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782656 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782666 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782676 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782685 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782696 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782706 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782716 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782725 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782734 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782743 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782752 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782761 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782770 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782780 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782790 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782799 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782810 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782819 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:53 UTC] USER=www-data EUID=0 PID=1782828 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:54 UTC] USER=www-data EUID=0 PID=1782837 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:42:54 UTC] USER=www-data EUID=0 PID=1782846 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:42:54 UTC] USER=www-data EUID=0 PID=1782855 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:42:54 UTC] USER=www-data EUID=0 PID=1782864 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:42:54 UTC] USER=www-data EUID=0 PID=1782873 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:42:54 UTC] USER=www-data EUID=0 PID=1782882 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:42:54 UTC] USER=www-data EUID=0 PID=1782892 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:42:54 UTC] USER=www-data EUID=0 PID=1782902 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:42:54 UTC] USER=www-data EUID=0 PID=1782911 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:42:54 UTC] USER=www-data EUID=0 PID=1782920 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:42:54 UTC] USER=www-data EUID=0 PID=1782929 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:42:54 UTC] USER=www-data EUID=0 PID=1782938 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:42:54 UTC] USER=www-data EUID=0 PID=1782947 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:42:54 UTC] USER=www-data EUID=0 PID=1782956 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:42:54 UTC] USER=www-data EUID=0 PID=1782965 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:42:54 UTC] USER=www-data EUID=0 PID=1782974 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:42:55 UTC] USER=www-data EUID=0 PID=1783015 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-01-02 08:42:55 UTC] USER=www-data EUID=0 PID=1783024 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 08:42:55 UTC] USER=www-data EUID=0 PID=1783033 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-01-02 08:42:55 UTC] USER=www-data EUID=0 PID=1783042 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 08:42:55 UTC] USER=www-data EUID=0 PID=1783051 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:42:55 UTC] USER=www-data EUID=0 PID=1783067 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:42:55 UTC] USER=www-data EUID=0 PID=1783076 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:42:55 UTC] USER=www-data EUID=0 PID=1783085 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:42:55 UTC] USER=www-data EUID=0 PID=1783095 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:42:55 UTC] USER=www-data EUID=0 PID=1783105 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:42:55 UTC] USER=www-data EUID=0 PID=1783114 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:42:55 UTC] USER=www-data EUID=0 PID=1783123 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783132 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783141 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783151 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783160 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783169 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783178 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783187 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783197 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783206 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783215 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783224 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783251 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783260 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783269 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783278 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783287 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783296 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783305 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783314 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783323 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783332 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783342 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783352 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783362 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:56 UTC] USER=www-data EUID=0 PID=1783371 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783380 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783389 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783398 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783407 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783416 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783425 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783434 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783443 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783452 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783462 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783472 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783481 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783490 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783499 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783509 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783518 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783528 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783537 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783548 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783557 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783566 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783576 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783586 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783595 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783604 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783613 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783622 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783631 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 08:42:57 UTC] USER=www-data EUID=0 PID=1783641 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 08:42:58 UTC] USER=www-data EUID=0 PID=1783651 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:42:58 UTC] USER=www-data EUID=0 PID=1783660 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:42:58 UTC] USER=www-data EUID=0 PID=1783669 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 08:42:58 UTC] USER=www-data EUID=0 PID=1783680 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 08:42:58 UTC] USER=www-data EUID=0 PID=1783690 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:42:58 UTC] USER=www-data EUID=0 PID=1783700 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:42:58 UTC] USER=www-data EUID=0 PID=1783709 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:42:58 UTC] USER=www-data EUID=0 PID=1783718 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:42:58 UTC] USER=www-data EUID=0 PID=1783727 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:42:58 UTC] USER=www-data EUID=0 PID=1783736 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:42:58 UTC] USER=www-data EUID=0 PID=1783745 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:42:58 UTC] USER=www-data EUID=0 PID=1783754 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:42:58 UTC] USER=www-data EUID=0 PID=1783763 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:42:58 UTC] USER=www-data EUID=0 PID=1783772 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-postgresql environment: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com (10.100.1.214)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.214
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01
[INFO] Data dir:   /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[INFO] Port:       5432
[INFO] Hostname:   db-identity-sau-main-dev-postgresql-worker-01
[2026-01-02 08:43:00 UTC] USER=www-data EUID=0 PID=1783875 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:43:00 UTC] USER=www-data EUID=0 PID=1783896 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:43:00 UTC] USER=www-data EUID=0 PID=1783917 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:43:00 UTC] USER=www-data EUID=0 PID=1783938 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 08:43:01 UTC] USER=www-data EUID=0 PID=1783980 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:43:01 UTC] USER=www-data EUID=0 PID=1783989 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-02 08:43:01 UTC] USER=www-data EUID=0 PID=1783999 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1783945
[2026-01-02 08:43:01 UTC] USER=www-data EUID=0 PID=1784011 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1783945/ra_root.crt
[2026-01-02 08:43:01 UTC] USER=www-data EUID=0 PID=1784029 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1783945/ra_root.key
[2026-01-02 08:43:01 UTC] USER=www-data EUID=0 PID=1784038 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1783945/ra_root.crt
[2026-01-02 08:43:01 UTC] USER=www-data EUID=0 PID=1784047 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1783945/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[2026-01-02 08:43:05 UTC] USER=www-data EUID=0 PID=1784108 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1783945/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 08:43:05 UTC] USER=www-data EUID=0 PID=1784117 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1783945/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 08:43:05 UTC] USER=www-data EUID=0 PID=1784126 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 08:43:05 UTC] USER=www-data EUID=0 PID=1784135 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1783945/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:43:05 UTC] USER=www-data EUID=0 PID=1784144 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:43:05 UTC] USER=www-data EUID=0 PID=1784153 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:43:05 UTC] USER=www-data EUID=0 PID=1784162 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 08:43:05 UTC] USER=www-data EUID=0 PID=1784173 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 08:43:05 UTC] USER=www-data EUID=0 PID=1784182 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 08:43:05 UTC] USER=www-data EUID=0 PID=1784191 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 08:43:05 UTC] USER=www-data EUID=0 PID=1784200 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 08:43:05 UTC] USER=www-data EUID=0 PID=1784209 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:43:05 UTC] USER=www-data EUID=0 PID=1784218 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01
Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-02 08:43:05 UTC] USER=www-data EUID=0 PID=1784247 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 08:43:05 UTC] USER=www-data EUID=0 PID=1784256 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 08:43:05 UTC] USER=www-data EUID=0 PID=1784265 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-02 08:43:05 UTC] USER=www-data EUID=0 PID=1784286 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-worker-01.service
[2026-01-02 08:43:06 UTC] USER=www-data EUID=0 PID=1784310 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-02 08:43:06 UTC] USER=www-data EUID=0 PID=1784341 ACTION=fsop ARGS=rm -rf /var/run/postgresql-identity-sau-main-dev-worker-01
[OK]   No conflicting Postgres left on port 5432
[OK]   Using postgres password from vault provider
[2026-01-02 08:43:09 UTC] USER=www-data EUID=0 PID=1784409 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.XTdvjD
[2026-01-02 08:43:09 UTC] USER=www-data EUID=0 PID=1784430 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.XTdvjD
[2026-01-02 08:43:09 UTC] USER=www-data EUID=0 PID=1784452 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 08:43:09 UTC] USER=www-data EUID=0 PID=1784474 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 08:43:09 UTC] USER=www-data EUID=0 PID=1784496 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/identity-sau-main-dev
[INFO] Initializing cluster in /var/lib/postgresql/17/identity-sau-main-dev/worker-01 (SCRAM; pwfile)
[WARN] Removing existing data directory: /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 08:43:09 UTC] USER=www-data EUID=0 PID=1784517 ACTION=fsop ARGS=rm -rf /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 08:43:09 UTC] USER=www-data EUID=0 PID=1784539 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 08:43:09 UTC] USER=www-data EUID=0 PID=1784560 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 08:43:10 UTC] USER=www-data EUID=0 PID=1784581 ACTION=fsop ARGS=chmod 700 /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 08:43:10 UTC] USER=www-data EUID=0 PID=1784602 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-02 08:43:10 UTC] USER=www-data EUID=0 PID=1784624 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-02 08:43:10 UTC] USER=www-data EUID=0 PID=1784647 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-02 08:43:10 UTC] USER=www-data EUID=0 PID=1784656 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /var/lib/postgresql/17/identity-sau-main-dev/worker-01 --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.XTdvjD
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/17/identity-sau-main-dev/worker-01 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /var/lib/postgresql/17/identity-sau-main-dev/worker-01 -l logfile start

[OK]   initdb complete
[2026-01-02 08:43:11 UTC] USER=www-data EUID=0 PID=1784696 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.XTdvjD
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-02 08:43:12 UTC] USER=www-data EUID=0 PID=1784745 ACTION=fsop ARGS=cp /tmp/tmp.6pgL3unii4 /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 08:43:12 UTC] USER=www-data EUID=0 PID=1784766 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 08:43:12 UTC] USER=www-data EUID=0 PID=1784787 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01.service
[2026-01-02 08:43:12 UTC] USER=www-data EUID=0 PID=1784814 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.hmDPct /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01.service
[2026-01-02 08:43:12 UTC] USER=www-data EUID=0 PID=1784835 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01.service
[OK]   systemd unit written
[2026-01-02 08:43:12 UTC] USER=www-data EUID=0 PID=1784856 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 08:43:12 UTC] USER=www-data EUID=0 PID=1784879 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 08:43:12 UTC] USER=www-data EUID=0 PID=1784902 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-02 08:43:14 UTC] USER=www-data EUID=0 PID=1785023 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-worker-01.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-02 08:43:15 UTC] USER=www-data EUID=0 PID=1785066 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-worker-01.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Provisioning application database and Debezium role (if not exists)...
[INFO] Checking if database fastorder_identity_sau_main_dev_db exists...
[INFO] DB check result: exit_code=0, output='[2026-01-02 08:43:16 UTC] USER=www-data EUID=0 PID=1785225 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_identity_sau_main_dev_db''
[INFO] Creating database fastorder_identity_sau_main_dev_db...
[2026-01-02 08:43:16 UTC] USER=www-data EUID=0 PID=1785248 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_identity_sau_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[OK]   Database fastorder_identity_sau_main_dev_db created
[INFO] Checking if role debezium_user exists...
[INFO] Role check result: exit_code=0, output='[2026-01-02 08:43:16 UTC] USER=www-data EUID=0 PID=1785272 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[INFO] Creating role debezium_user...
[2026-01-02 08:43:17 UTC] USER=www-data EUID=0 PID=1785299 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'TFSfgPNoeCiDz7kKv70Tpbql';
CREATE ROLE
[OK]   Role debezium_user created
[2026-01-02 08:43:17 UTC] USER=www-data EUID=0 PID=1785324 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_identity_sau_main_dev_db" TO debezium_user;
GRANT
[OK]   Application DB (fastorder_identity_sau_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[INFO] Applying connection and memory optimizations...
[INFO] Current settings: max_connections=100, work_mem=4MB
[INFO] Target settings (worker): max_connections=100, work_mem=8MB
[2026-01-02 08:43:17 UTC] USER=www-data EUID=0 PID=1785405 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 100;
ALTER SYSTEM
[2026-01-02 08:43:18 UTC] USER=www-data EUID=0 PID=1785430 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-02 08:43:18 UTC] USER=www-data EUID=0 PID=1785454 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   Settings applied to postgresql.auto.conf
[2026-01-02 08:43:18 UTC] USER=www-data EUID=0 PID=1785469 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01/standby.signal
[INFO] Service recently started (3s ago) - restarting to apply max_connections...
[INFO] Stopping service...
[2026-01-02 08:43:18 UTC] USER=www-data EUID=0 PID=1785491 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-worker-01.service
[INFO] Waiting for port 5432 to be released...
[OK]   Port 5432 released
[INFO] Starting service...
[2026-01-02 08:43:21 UTC] USER=www-data EUID=0 PID=1785537 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-worker-01.service
[2026-01-02 08:43:27 UTC] USER=www-data EUID=0 PID=1785800 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-worker-01.service
[OK]   ✅ Optimization complete: max_connections=100, work_mem=8MB
[OK]   Synchronous replication already configured (synchronous_commit: on)
[INFO] Setting postgres password via centralized script... for worker-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Initial setup: Using password from initdb
✓ PostgreSQL password already set during initdb
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/worker-01
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
[OK]   Password set and persisted
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.214
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01...
[INFO]   db-identity-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.214

[INFO]   ✅ db-identity-sau-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.214    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com


[OK]   PostgreSQL 'identity-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key \
        host=db-identity-sau-main-dev-postgresql-worker-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-worker-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.214
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-worker-01
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 2a8d7237-0c1b-4286-8ffc-cd46f4f7052e
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 📦 03 role...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 08:43:42 UTC] USER=www-data EUID=0 PID=1786320 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    debezium_user
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   debezium_user
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:44:06 UTC] USER=www-data EUID=0 PID=1786562 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-debezium_user
[2026-01-02 08:44:06 UTC] USER=www-data EUID=0 PID=1786571 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-01-02 08:44:06 UTC] USER=www-data EUID=0 PID=1786580 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-debezium_user/ra_root.key
[2026-01-02 08:44:06 UTC] USER=www-data EUID=0 PID=1786590 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-01-02 08:44:06 UTC] USER=www-data EUID=0 PID=1786599 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:44:07 UTC] USER=www-data EUID=0 PID=1786616 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:44:07 UTC] USER=www-data EUID=0 PID=1786630 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:44:07 UTC] USER=www-data EUID=0 PID=1786640 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 08:44:07 UTC] USER=www-data EUID=0 PID=1786649 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 08:44:07 UTC] USER=www-data EUID=0 PID=1786658 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:07 UTC] USER=www-data EUID=0 PID=1786667 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:44:07 UTC] USER=www-data EUID=0 PID=1786676 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 08:44:07 UTC] USER=www-data EUID=0 PID=1786685 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 08:44:07 UTC] USER=www-data EUID=0 PID=1786696 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 08:44:07 UTC] USER=www-data EUID=0 PID=1786705 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:07 UTC] USER=www-data EUID=0 PID=1786714 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:44:07 UTC] USER=www-data EUID=0 PID=1786723 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 08:44:07 UTC] USER=www-data EUID=0 PID=1786732 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 08:44:07 UTC] USER=www-data EUID=0 PID=1786742 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 08:44:07 UTC] USER=www-data EUID=0 PID=1786751 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:07 UTC] USER=www-data EUID=0 PID=1786760 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786786 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786795 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786806 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786815 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786824 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786834 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786843 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786858 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786868 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786877 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786886 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786896 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786906 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786915 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786924 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786933 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786942 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786952 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786965 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786976 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786986 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1786995 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 08:44:08 UTC] USER=www-data EUID=0 PID=1787004 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787014 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787024 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787033 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787042 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787051 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787060 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787069 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787080 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787090 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787099 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787108 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787117 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787128 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787138 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787147 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787156 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787165 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787174 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787183 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787192 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787201 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787212 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787221 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787230 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787240 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787250 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:44:09 UTC] USER=www-data EUID=0 PID=1787259 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:44:10 UTC] USER=www-data EUID=0 PID=1787268 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:44:10 UTC] USER=www-data EUID=0 PID=1787277 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:44:10 UTC] USER=www-data EUID=0 PID=1787286 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:44:10 UTC] USER=www-data EUID=0 PID=1787295 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:44:10 UTC] USER=www-data EUID=0 PID=1787304 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:44:10 UTC] USER=www-data EUID=0 PID=1787313 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:44:10 UTC] USER=www-data EUID=0 PID=1787324 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: debezium_user
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U debezium_user -d postgres

🔐 Generating replicator client certificate for worker-01...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:44:10 UTC] USER=www-data EUID=0 PID=1787365 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-02 08:44:10 UTC] USER=www-data EUID=0 PID=1787374 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 08:44:10 UTC] USER=www-data EUID=0 PID=1787383 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-02 08:44:10 UTC] USER=www-data EUID=0 PID=1787392 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 08:44:10 UTC] USER=www-data EUID=0 PID=1787401 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787415 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787424 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787433 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787442 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787451 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787460 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787469 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787478 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787487 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787497 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787506 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787515 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787524 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787533 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787542 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787551 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787560 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787569 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787597 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787606 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787615 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787624 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:11 UTC] USER=www-data EUID=0 PID=1787633 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787642 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787651 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787660 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787669 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787678 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787688 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787699 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787709 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787718 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787727 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787736 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787745 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787754 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787763 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787772 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787781 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787790 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787799 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787809 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787821 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787830 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787840 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787849 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787858 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787867 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787876 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:44:12 UTC] USER=www-data EUID=0 PID=1787885 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1787905 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1787914 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1787925 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1787935 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1787944 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1787953 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1787962 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1787971 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1787980 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1787989 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1787998 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1788007 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1788018 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1788027 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1788037 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1788047 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1788056 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1788065 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:44:13 UTC] USER=www-data EUID=0 PID=1788074 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:44:14 UTC] USER=www-data EUID=0 PID=1788083 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:44:14 UTC] USER=www-data EUID=0 PID=1788092 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:44:14 UTC] USER=www-data EUID=0 PID=1788101 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:44:14 UTC] USER=www-data EUID=0 PID=1788110 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:44:14 UTC] USER=www-data EUID=0 PID=1788119 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

✅ Replicator certificate generated for worker-01
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: fastorder_admin_gd
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: worker-01/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    fastorder_admin_gd
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   fastorder_admin_gd
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:44:30 UTC] USER=www-data EUID=0 PID=1788727 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-fastorder_admin_gd
[2026-01-02 08:44:30 UTC] USER=www-data EUID=0 PID=1788736 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-01-02 08:44:30 UTC] USER=www-data EUID=0 PID=1788745 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
[2026-01-02 08:44:31 UTC] USER=www-data EUID=0 PID=1788754 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-01-02 08:44:31 UTC] USER=www-data EUID=0 PID=1788763 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:44:31 UTC] USER=www-data EUID=0 PID=1788780 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:44:31 UTC] USER=www-data EUID=0 PID=1788789 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:44:31 UTC] USER=www-data EUID=0 PID=1788798 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 08:44:31 UTC] USER=www-data EUID=0 PID=1788807 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 08:44:31 UTC] USER=www-data EUID=0 PID=1788816 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:31 UTC] USER=www-data EUID=0 PID=1788825 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:44:31 UTC] USER=www-data EUID=0 PID=1788834 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:44:31 UTC] USER=www-data EUID=0 PID=1788843 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 08:44:31 UTC] USER=www-data EUID=0 PID=1788852 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 08:44:31 UTC] USER=www-data EUID=0 PID=1788863 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:44:31 UTC] USER=www-data EUID=0 PID=1788872 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 08:44:31 UTC] USER=www-data EUID=0 PID=1788882 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:31 UTC] USER=www-data EUID=0 PID=1788894 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:44:31 UTC] USER=www-data EUID=0 PID=1788934 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1788963 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1788994 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789024 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789036 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789065 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789074 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789083 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789092 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789101 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789110 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789119 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789128 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789137 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789146 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789155 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789165 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789177 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789186 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789195 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789204 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789213 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789222 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789231 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789240 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789249 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:44:32 UTC] USER=www-data EUID=0 PID=1789258 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789267 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789277 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789287 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789298 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789307 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789316 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789325 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789334 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789343 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789352 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789361 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789370 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789379 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789389 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789399 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789408 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789417 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789426 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789435 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789444 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789453 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789464 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789473 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789482 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789491 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789501 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789511 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:44:33 UTC] USER=www-data EUID=0 PID=1789520 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:44:34 UTC] USER=www-data EUID=0 PID=1789529 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:44:34 UTC] USER=www-data EUID=0 PID=1789538 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:44:34 UTC] USER=www-data EUID=0 PID=1789547 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:44:34 UTC] USER=www-data EUID=0 PID=1789556 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:44:34 UTC] USER=www-data EUID=0 PID=1789565 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:44:34 UTC] USER=www-data EUID=0 PID=1789577 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:44:34 UTC] USER=www-data EUID=0 PID=1789592 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: fastorder_admin_gd
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-identity-sau-main-dev-worker-01:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_identity_sau_main_dev_db already exists, skipping creation
[2026-01-02 08:44:34 UTC] USER=www-data EUID=0 PID=1789667 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-02 08:44:34 UTC] USER=www-data EUID=0 PID=1789701 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

=== Pre-flight Checks ===
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
✓ AWS Secrets Manager accessible

=== Retrieving Credentials from AWS ===
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
/opt/fastorder/bash/infra_core/cache.sh: line 145: /var/cache/secrets/fastorder_db_identity_sau_main_dev_postgresql_worker-01_fastorder_admin_gd.cache.tmp.1789724: Permission denied
✅ Retrieved from secrets manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials loaded for worker-01/fastorder_admin_gd: fastorder_admin_gd@db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_identity_sau_main_dev_db
✓ Credentials retrieved: fastorder_admin_gd@db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_identity_sau_main_dev_db
╔════════════════════════════════════════════╗
║  PostgreSQL Test Suite (AWS Secrets MGR)  ║
╚════════════════════════════════════════════╝

=== PostgreSQL Authentication Test ===
✗ PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-identity-sau-main-dev-postgresql-worker-01.fastorder.com" (10.100.1.214), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 08:44:43 UTC] USER=www-data EUID=0 PID=1789920 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_identity_sau_main_dev_db
  SCHEMA      : auth
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
  DNS → 10.100.1.214
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User debezium_user does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: debezium_user
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: worker-01/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-01-02 08:44:58 UTC] USER=www-data EUID=0 PID=1790265 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
[2026-01-02 08:44:58 UTC] USER=www-data EUID=0 PID=1790291 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d fastorder_identity_sau_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
🔐 Creating replicator role for worker-01...
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
  DNS → 10.100.1.214
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Creating role: replicator with password
SET
CREATE ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/identity/sau/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🎉 Done.
✅ Replicator role created for worker-01

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=worker-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║                    IAM Database Schema Initialization                       ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] 🟢 Starting IAM schema provisioning...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

[INFO] 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: core
  Core Identity Directory (tenants, realms, identities, devices, MFA)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [1/20]: core/01-tenant
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "pgcrypto" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "citext" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
NOTICE:  schema "utils" already exists, skipping
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
NOTICE:  schema "core" already exists, skipping
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
NOTICE:  relation "tenant" already exists, skipping
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[OK] Table core/01-tenant initialized

[INFO] 🔸 Table [2/20]: core/02-realm
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  relation "realm" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_realm_keycloak_id" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_realm_tenant" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[OK] Table core/02-realm initialized

[INFO] 🔸 Table [3/20]: core/03-identity
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  relation "identity" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_unique_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_unique_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_type" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[OK] Table core/03-identity initialized

[INFO] 🔸 Table [4/20]: core/04-device
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  relation "device" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_device_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_fingerprint" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_trusted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_last_seen" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[OK] Table core/04-device initialized

[INFO] 🔸 Table [5/20]: core/05-identity_account
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  relation "identity_account" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_account_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_lockout" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_last_login" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[OK] Table core/05-identity_account initialized

[INFO] 🔸 Table [6/20]: core/06-identity_mfa
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  relation "identity_mfa" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_mfa_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_active" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[OK] Table core/06-identity_mfa initialized

[INFO] 🔸 Table [7/20]: core/07-external_idp_link
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  relation "external_idp_link" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_external_idp_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_provider" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_email" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[OK] Table core/07-external_idp_link initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: policy
  RBAC/ABAC Authorization (clients, roles, permissions, policies)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [8/20]: policy/01-client
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
NOTICE:  schema "policy" already exists, skipping
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  relation "client" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_client_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_key" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_status" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[OK] Table policy/01-client initialized

[INFO] 🔸 Table [9/20]: policy/02-resource
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  relation "resource" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_resource_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_external" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_owner" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[OK] Table policy/02-resource initialized

[INFO] 🔸 Table [10/20]: policy/03-scope
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  relation "scope" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_scope_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_scope_name" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[OK] Table policy/03-scope initialized

[INFO] 🔸 Table [11/20]: policy/04-permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  relation "permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_permission_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_resource" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[OK] Table policy/04-permission initialized

[INFO] 🔸 Table [12/20]: policy/05-role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  relation "role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_keycloak" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[OK] Table policy/05-role initialized

[INFO] 🔸 Table [13/20]: policy/06-role_permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  relation "role_permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_permission_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_permission_perm" already exists, skipping
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[OK] Table policy/06-role_permission initialized

[INFO] 🔸 Table [14/20]: policy/07-identity_role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  relation "identity_role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_role_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_active" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[OK] Table policy/07-identity_role initialized

[INFO] 🔸 Table [15/20]: policy/08-policy_rule
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  relation "policy_rule" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_policy_rule_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_enabled" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_priority" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[OK] Table policy/08-policy_rule initialized

[INFO] 🔸 Table [16/20]: policy/09-api_key
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  relation "api_key" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_api_key_prefix" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[OK] Table policy/09-api_key initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: audit
  Audit & Risk Logging (auth events, admin actions, risk decisions)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [17/20]: audit/01-auth_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
NOTICE:  schema "audit" already exists, skipping
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
NOTICE:  relation "auth_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_auth_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_result" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_ip" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_session" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_trace" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_risk" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[OK] Table audit/01-auth_event initialized

[INFO] 🔸 Table [18/20]: audit/02-admin_action
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
NOTICE:  relation "admin_action" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_admin_action_actor" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_target" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_trace" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[OK] Table audit/02-admin_action initialized

[INFO] 🔸 Table [19/20]: audit/03-risk_decision
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
NOTICE:  relation "risk_decision" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_risk_decision_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_level" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_decision" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_auth" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[OK] Table audit/03-risk_decision initialized

[INFO] 🔸 Table [20/20]: audit/04-consent_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
NOTICE:  relation "consent_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_consent_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_version" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_granted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  relation "audit.auth_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  relation "audit.auth_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  relation "audit.admin_action_2026_03" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  relation "audit.admin_action_2026_04" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  relation "audit.risk_decision_2026_03" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  relation "audit.risk_decision_2026_04" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  relation "audit.consent_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  relation "audit.consent_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[OK] Table audit/04-consent_event initialized


════════════════════════════════════════════════════════════════════════════
[OK] ✅ IAM Schema Initialization Complete!
[OK] All 20 tables initialized successfully

Schemas created:
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

Design highlights:
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

════════════════════════════════════════════════════════════════════════════

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=worker-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: identity/run.sh started for SERVICE=identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity
🔍 DEBUG_CHECKPOINT_A3: SERVICE_ROOT exists, discovering table folders
🔍 DEBUG_CHECKPOINT_A4: Found subfolder: auth
🔍 DEBUG_CHECKPOINT_A4b: Checking for nested schema layout in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth
🔍 DEBUG_CHECKPOINT_A4c: Found nested steps dir: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps (display: auth/login)
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: auth/login|/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
[INFO] 📚 Detected grouped table folders under identity/: auth/login

🔍 DEBUG_CHECKPOINT_A7: Current IDENTIFIER=coordinator
🔍 DEBUG_CHECKPOINT_A8_PROCEED: Processing tables on coordinator/main node
🔍 DEBUG_CHECKPOINT_A9: Processing table: auth/login at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
[INFO] 🔸 Table group: auth/login
🔍 DEBUG_CHECKPOINT_A10: About to run numbered steps for table: auth/login
🔍 DEBUG_CHECKPOINT_B1: run_all_numbered_steps_in_dir called for dir=/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps table=auth/login
🔍 DEBUG_CHECKPOINT_B2: Found 1 numbered steps: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B3: About to run step: 01-init-schema.sh
Ab substep 0 compelete start
[DEBUG] Tracking substep start: steps/01-install/steps/identity/auth/login/01-init-schema (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
Ab substep 0 compelete start
[INFO] 📦 01 init schema...
Ab substep 1 compelete start
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing auth.login_account table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating auth schema...
NOTICE:  schema "auth" already exists, skipping
CREATE SCHEMA
✅ Schema created
🔧 Creating account_status ENUM...
DO
✅ ENUM created
🔧 Creating auth.login_account table...
NOTICE:  relation "login_account" already exists, skipping
CREATE TABLE
✅ Table created (Citus-compatible with region_hint in all constraints)
🔧 Creating indexes...
NOTICE:  relation "idx_login_account_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_login_account_username" already exists, skipping
CREATE INDEX
✅ Indexes created
ℹ️  Table already registered with Citus
🎉 Schema initialization complete for fastorder_identity_sau_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

📊 Registering environment in monitoring database (obs schema)...
   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
   Resource IP: 142.93.238.16
⚠️  Could not connect to monitoring database, skipping registration
   You can manually register later using:
   /opt/fastorder/bash/scripts/env_app_setup/setup/04-postgresql/steps/register-authN-af-aaaa1-dev.sh

==========================================
✅ Schema initialization complete!
==========================================
Ab substep 1 compelete end
Ab substep 2 compelete start
Ab substep 2 compelete end

🔍 DEBUG_CHECKPOINT_B4: Completed step: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B5: All numbered steps completed for /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A11: Completed numbered steps for table: auth/login
compeleted here

🔍 DEBUG_CHECKPOINT_A12: All tables processed
End of 04-postgresql/steps/01-install/steps/identity/run.sh

✓ ✅ Worker worker-01 setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up standby replicas (1 per worker)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up standby: worker-01-standby-01 (replica of worker-01)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-02 08:55:08 UTC] USER=unknown EUID=33 PID=1801295 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 08:55:08 UTC] USER=unknown EUID=33 PID=1801302 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 08:55:08 UTC] USER=unknown EUID=33 PID=1801309 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-02 08:55:08 UTC] USER=unknown EUID=33 PID=1801316 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 08:55:08 UTC] USER=unknown EUID=33 PID=1801325 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-02 08:55:08 UTC] USER=unknown EUID=33 PID=1801332 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4 JOB_UUID=ccddb67e-4182-4074-b2a8-add87af57fe4

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] PostgreSQL IP: 10.100.1.211
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01-standby-01...
[INFO]   db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.211

[INFO]   ✅ db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.211    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  identity-sau-main-dev.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 08:55:11 UTC] USER=www-data EUID=0 PID=1801459 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:11 UTC] USER=www-data EUID=0 PID=1801468 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-02 08:55:11 UTC] USER=www-data EUID=0 PID=1801481 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1801425
[2026-01-02 08:55:11 UTC] USER=www-data EUID=0 PID=1801490 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1801425/ra_root.crt
[2026-01-02 08:55:12 UTC] USER=www-data EUID=0 PID=1801499 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1801425/ra_root.key
[2026-01-02 08:55:12 UTC] USER=www-data EUID=0 PID=1801508 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1801425/ra_root.crt
[2026-01-02 08:55:12 UTC] USER=www-data EUID=0 PID=1801517 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1801425/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
[2026-01-02 08:55:16 UTC] USER=www-data EUID=0 PID=1801569 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1801425/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 08:55:16 UTC] USER=www-data EUID=0 PID=1801578 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1801425/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 08:55:16 UTC] USER=www-data EUID=0 PID=1801587 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 08:55:16 UTC] USER=www-data EUID=0 PID=1801597 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1801425/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:16 UTC] USER=www-data EUID=0 PID=1801606 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:16 UTC] USER=www-data EUID=0 PID=1801615 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:16 UTC] USER=www-data EUID=0 PID=1801624 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 08:55:16 UTC] USER=www-data EUID=0 PID=1801635 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 08:55:16 UTC] USER=www-data EUID=0 PID=1801644 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 08:55:16 UTC] USER=www-data EUID=0 PID=1801653 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 08:55:16 UTC] USER=www-data EUID=0 PID=1801662 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 08:55:16 UTC] USER=www-data EUID=0 PID=1801671 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:16 UTC] USER=www-data EUID=0 PID=1801680 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:identity-sau-main-dev.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01-standby-01
Primary CN:  identity-sau-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=identity-sau-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801736 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801745 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801754 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801765 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801774 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801788 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801797 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801806 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801815 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801824 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801833 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801842 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801851 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801860 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801869 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801878 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801887 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:55:17 UTC] USER=www-data EUID=0 PID=1801896 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1801905 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1801914 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1801923 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1801949 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1801958 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1801967 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1801976 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1801985 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1801994 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802003 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802012 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802021 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802030 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802039 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802049 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802059 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802068 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802077 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802086 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802095 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802104 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802113 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802122 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802131 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802140 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802149 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802161 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:18 UTC] USER=www-data EUID=0 PID=1802171 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802180 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802189 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802198 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802207 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802217 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802226 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802239 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802265 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802297 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802323 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802369 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802382 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802391 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802401 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802410 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802419 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802428 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802438 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802447 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802456 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802465 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802474 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802486 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802500 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:55:19 UTC] USER=www-data EUID=0 PID=1802531 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:55:20 UTC] USER=www-data EUID=0 PID=1802560 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:55:20 UTC] USER=www-data EUID=0 PID=1802593 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:55:20 UTC] USER=www-data EUID=0 PID=1802627 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:55:20 UTC] USER=www-data EUID=0 PID=1802642 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:55:20 UTC] USER=www-data EUID=0 PID=1802651 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:55:20 UTC] USER=www-data EUID=0 PID=1802661 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:55:20 UTC] USER=www-data EUID=0 PID=1802670 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:55:20 UTC] USER=www-data EUID=0 PID=1802713 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-01-02 08:55:20 UTC] USER=www-data EUID=0 PID=1802722 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 08:55:20 UTC] USER=www-data EUID=0 PID=1802731 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
[2026-01-02 08:55:20 UTC] USER=www-data EUID=0 PID=1802741 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 08:55:20 UTC] USER=www-data EUID=0 PID=1802751 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:21 UTC] USER=www-data EUID=0 PID=1802767 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:21 UTC] USER=www-data EUID=0 PID=1802776 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:21 UTC] USER=www-data EUID=0 PID=1802785 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:55:21 UTC] USER=www-data EUID=0 PID=1802794 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:55:21 UTC] USER=www-data EUID=0 PID=1802803 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:21 UTC] USER=www-data EUID=0 PID=1802812 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:55:21 UTC] USER=www-data EUID=0 PID=1802821 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:55:21 UTC] USER=www-data EUID=0 PID=1802830 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:55:21 UTC] USER=www-data EUID=0 PID=1802839 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:55:21 UTC] USER=www-data EUID=0 PID=1802848 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:55:21 UTC] USER=www-data EUID=0 PID=1802857 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:55:21 UTC] USER=www-data EUID=0 PID=1802866 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:21 UTC] USER=www-data EUID=0 PID=1802875 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:21 UTC] USER=www-data EUID=0 PID=1802884 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:55:21 UTC] USER=www-data EUID=0 PID=1802893 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:55:21 UTC] USER=www-data EUID=0 PID=1802903 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:55:21 UTC] USER=www-data EUID=0 PID=1802912 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1802921 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1802947 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1802956 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1802965 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1802974 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1802983 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1802996 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803026 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803053 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803081 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803116 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803136 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803146 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803157 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803166 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803175 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803184 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803193 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803202 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803211 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803220 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803229 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803238 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803250 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:55:22 UTC] USER=www-data EUID=0 PID=1803260 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803270 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803279 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803297 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803306 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803315 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803324 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803333 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803352 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803363 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803373 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803384 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803393 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803402 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803411 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803420 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803429 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803438 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803447 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803456 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803465 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803476 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803486 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803496 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:55:23 UTC] USER=www-data EUID=0 PID=1803505 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:55:24 UTC] USER=www-data EUID=0 PID=1803514 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:55:24 UTC] USER=www-data EUID=0 PID=1803523 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:55:24 UTC] USER=www-data EUID=0 PID=1803532 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:55:24 UTC] USER=www-data EUID=0 PID=1803541 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:55:24 UTC] USER=www-data EUID=0 PID=1803550 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:55:24 UTC] USER=www-data EUID=0 PID=1803559 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:55:24 UTC] USER=www-data EUID=0 PID=1803568 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:55:24 UTC] USER=www-data EUID=0 PID=1803612 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-02 08:55:24 UTC] USER=www-data EUID=0 PID=1803621 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 08:55:24 UTC] USER=www-data EUID=0 PID=1803630 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-02 08:55:24 UTC] USER=www-data EUID=0 PID=1803641 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 08:55:24 UTC] USER=www-data EUID=0 PID=1803650 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803664 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803673 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803682 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803691 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803700 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803711 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803720 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803729 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803738 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803747 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803756 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803765 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803774 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803783 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803792 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803801 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803810 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:55:25 UTC] USER=www-data EUID=0 PID=1803819 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1803847 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1803856 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1803865 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1803874 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1803883 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1803892 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1803901 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1803910 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1803919 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1803928 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1803937 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1803949 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1803959 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1803968 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1803977 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1803986 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1803995 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1804004 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1804013 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1804022 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1804031 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1804040 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1804049 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1804059 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1804072 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1804089 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1804102 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1804112 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1804126 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:55:26 UTC] USER=www-data EUID=0 PID=1804135 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804146 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804155 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804164 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804173 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804183 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804193 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804203 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804212 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804221 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804230 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804240 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804249 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804258 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804267 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804276 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804285 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804294 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804304 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804314 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804323 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804332 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804341 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804350 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804359 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804368 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804377 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:55:27 UTC] USER=www-data EUID=0 PID=1804386 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-standby-01-postgresql environment: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com (10.100.1.211)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.211
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] Data dir:   /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[INFO] Port:       5432
[INFO] Hostname:   db-identity-sau-main-dev-postgresql-worker-01-standby-01
[2026-01-02 08:55:29 UTC] USER=www-data EUID=0 PID=1804484 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:29 UTC] USER=www-data EUID=0 PID=1804505 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:29 UTC] USER=www-data EUID=0 PID=1804531 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:30 UTC] USER=www-data EUID=0 PID=1804552 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  identity-sau-main-dev.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 08:55:30 UTC] USER=www-data EUID=0 PID=1804597 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:30 UTC] USER=www-data EUID=0 PID=1804606 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-02 08:55:30 UTC] USER=www-data EUID=0 PID=1804616 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1804559
[2026-01-02 08:55:30 UTC] USER=www-data EUID=0 PID=1804625 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1804559/ra_root.crt
[2026-01-02 08:55:30 UTC] USER=www-data EUID=0 PID=1804634 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1804559/ra_root.key
[2026-01-02 08:55:30 UTC] USER=www-data EUID=0 PID=1804643 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1804559/ra_root.crt
[2026-01-02 08:55:30 UTC] USER=www-data EUID=0 PID=1804652 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1804559/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
[2026-01-02 08:55:33 UTC] USER=www-data EUID=0 PID=1804699 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1804559/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 08:55:33 UTC] USER=www-data EUID=0 PID=1804708 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1804559/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 08:55:33 UTC] USER=www-data EUID=0 PID=1804717 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 08:55:33 UTC] USER=www-data EUID=0 PID=1804726 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1804559/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:33 UTC] USER=www-data EUID=0 PID=1804735 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:33 UTC] USER=www-data EUID=0 PID=1804744 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 08:55:33 UTC] USER=www-data EUID=0 PID=1804753 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 08:55:33 UTC] USER=www-data EUID=0 PID=1804764 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 08:55:33 UTC] USER=www-data EUID=0 PID=1804773 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 08:55:33 UTC] USER=www-data EUID=0 PID=1804785 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 08:55:33 UTC] USER=www-data EUID=0 PID=1804794 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 08:55:34 UTC] USER=www-data EUID=0 PID=1804803 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:55:34 UTC] USER=www-data EUID=0 PID=1804812 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:identity-sau-main-dev.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01-standby-01
Primary CN:  identity-sau-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=identity-sau-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-02 08:55:34 UTC] USER=www-data EUID=0 PID=1804841 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 08:55:34 UTC] USER=www-data EUID=0 PID=1804850 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 08:55:34 UTC] USER=www-data EUID=0 PID=1804859 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-02 08:55:34 UTC] USER=www-data EUID=0 PID=1804883 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-worker-01-standby-01.service
[2026-01-02 08:55:34 UTC] USER=www-data EUID=0 PID=1804909 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[2026-01-02 08:55:34 UTC] USER=www-data EUID=0 PID=1804942 ACTION=fsop ARGS=rm -rf /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[OK]   No conflicting Postgres left on port 5432
[OK]   Generated new postgres password for initdb
[2026-01-02 08:55:59 UTC] USER=www-data EUID=0 PID=1805125 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.Qv2JEO
[2026-01-02 08:55:59 UTC] USER=www-data EUID=0 PID=1805147 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.Qv2JEO
[2026-01-02 08:56:00 UTC] USER=www-data EUID=0 PID=1805169 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 08:56:00 UTC] USER=www-data EUID=0 PID=1805191 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 08:56:00 UTC] USER=www-data EUID=0 PID=1805213 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/identity-sau-main-dev
[INFO] This is a standby. Using pg_basebackup from primary (worker-01)...
[INFO] Setting up replicator role and slot on primary (worker-01)...
ℹ️  Scanning primary for stuck queries from previous failed attempts...
ℹ️  Scanning for stuck queries (timeout: 30s)...
ℹ️  No stuck queries found
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SLOT        : worker_01_standby_01
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
  DNS → 10.100.1.214
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Role replicator already exists, updating password and ensuring REPLICATION privilege
SET
ALTER ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/identity/sau/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🔧 Ensuring replication slot: worker_01_standby_01…
🆕 Creating replication slot worker_01_standby_01
SET
 pg_create_physical_replication_slot 
-------------------------------------
 (worker_01_standby_01,)
(1 row)

✅ Replication slot worker_01_standby_01 created.
🎉 Done.
[OK]   Replicator role and slot created on primary
[INFO] Creating replicator client certificates for connecting to primary (worker-01)...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:56:04 UTC] USER=www-data EUID=0 PID=1805376 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-02 08:56:04 UTC] USER=www-data EUID=0 PID=1805385 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 08:56:04 UTC] USER=www-data EUID=0 PID=1805396 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-02 08:56:04 UTC] USER=www-data EUID=0 PID=1805405 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 08:56:04 UTC] USER=www-data EUID=0 PID=1805414 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:56:05 UTC] USER=www-data EUID=0 PID=1805430 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:56:05 UTC] USER=www-data EUID=0 PID=1805439 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:56:05 UTC] USER=www-data EUID=0 PID=1805450 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:56:05 UTC] USER=www-data EUID=0 PID=1805459 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:56:05 UTC] USER=www-data EUID=0 PID=1805468 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:56:05 UTC] USER=www-data EUID=0 PID=1805477 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:56:05 UTC] USER=www-data EUID=0 PID=1805486 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:56:05 UTC] USER=www-data EUID=0 PID=1805495 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:56:05 UTC] USER=www-data EUID=0 PID=1805504 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:56:05 UTC] USER=www-data EUID=0 PID=1805513 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:56:05 UTC] USER=www-data EUID=0 PID=1805522 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:56:05 UTC] USER=www-data EUID=0 PID=1805531 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:56:05 UTC] USER=www-data EUID=0 PID=1805540 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 08:56:05 UTC] USER=www-data EUID=0 PID=1805549 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:56:05 UTC] USER=www-data EUID=0 PID=1805558 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:56:05 UTC] USER=www-data EUID=0 PID=1805568 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:56:05 UTC] USER=www-data EUID=0 PID=1805577 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805586 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805612 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805621 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805630 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805639 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805648 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805657 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805666 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805676 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805685 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805694 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805703 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805713 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805723 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805732 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805741 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805750 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805759 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805768 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805777 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805786 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805795 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805804 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805813 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805823 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805835 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805845 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:56:06 UTC] USER=www-data EUID=0 PID=1805854 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1805872 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1805881 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1805890 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1805899 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1805908 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1805917 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1805926 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1805937 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1805949 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1805958 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1805967 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1805976 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1805985 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1805994 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1806003 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1806012 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1806021 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1806030 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1806039 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1806049 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1806061 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1806070 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1806079 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1806088 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:56:07 UTC] USER=www-data EUID=0 PID=1806106 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:56:08 UTC] USER=www-data EUID=0 PID=1806115 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:56:08 UTC] USER=www-data EUID=0 PID=1806124 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:56:08 UTC] USER=www-data EUID=0 PID=1806133 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

[OK]   Replicator certificate created for worker-01 in /home/postgres/
[INFO] Using replicator certificates from primary worker-01...
[2026-01-02 08:56:08 UTC] USER=www-data EUID=0 PID=1806161 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 08:56:08 UTC] USER=www-data EUID=0 PID=1806184 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 08:56:08 UTC] USER=www-data EUID=0 PID=1806205 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[OK]   Replicator certificates verified at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[OK]   root.crt verified at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[INFO] Updating primary pg_hba.conf to allow replication...
[INFO]   Standby IP: 10.100.1.211/32 (standby's source IP)
[INFO]   Primary application IP: 10.100.1.214/32 (for local pg_basebackup)
[INFO]   Primary DNS IP: 10.100.1.214/32 (DNS resolution of db-identity-sau-main-dev-postgresql-worker-01.fastorder.com)
[2026-01-02 08:56:08 UTC] USER=www-data EUID=0 PID=1806235 ACTION=passthru ARGS=grep -qxF # BEGIN standby-replication (managed) /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 08:56:08 UTC] USER=www-data EUID=0 PID=1806280 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.211/32  scram-sha-256 
      $0==begin {inside=1}
      inside && $0==rule {found=1}
      $0==end {inside=0}
      END {exit found?0:1}
     /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 08:56:08 UTC] USER=www-data EUID=0 PID=1806304 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl  replication  replicator  10.100.1.211/32  scram-sha-256 /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 08:56:09 UTC] USER=www-data EUID=0 PID=1806326 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.214/32  scram-sha-256 
        $0==begin {inside=1}
        inside && $0==rule {found=1}
        $0==end {inside=0}
        END {exit found?0:1}
       /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 08:56:09 UTC] USER=www-data EUID=0 PID=1806350 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl  replication  replicator  10.100.1.214/32  scram-sha-256 /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[INFO] Reloading primary PostgreSQL service...
[2026-01-02 08:56:09 UTC] USER=www-data EUID=0 PID=1806371 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
[OK]   Primary pg_hba.conf updated and service reloaded
[WARN] Removing existing data directory: /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 08:56:09 UTC] USER=www-data EUID=0 PID=1806393 ACTION=fsop ARGS=rm -rf /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[INFO] Primary host: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO] Using replicator cert: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[INFO] Using replicator key: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key (PKCS#8 format)
[INFO] Using CA cert: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[INFO] Verifying postgres user can access certificates...
[ERR]  postgres user CANNOT read /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[INFO] File permissions:
lrwxrwxrwx 1 postgres ssl-cert 72 Jan  2 08:56 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt -> /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[INFO] Parent directory permissions:
drwx------ 2 postgres postgres 4096 Jan  2 08:56 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
drwx------ 6 postgres postgres 4096 Jan  2 07:10 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[WARN] Attempting to fix permissions (/usr/local/bin/fastorder-provisioning-wrapper.sh required)...
[INFO] Fixing /home/postgres/ directory...
[2026-01-02 08:56:09 UTC] USER=www-data EUID=0 PID=1806460 ACTION=fsop ARGS=chmod 755 /home/postgres/
[INFO] Fixing /home/postgres/ssl/.postgresql/...
[2026-01-02 08:56:10 UTC] USER=www-data EUID=0 PID=1806483 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/
[INFO] Fixing parent directory: /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:56:10 UTC] USER=www-data EUID=0 PID=1806506 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[INFO] Fixing certificate directory: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 08:56:10 UTC] USER=www-data EUID=0 PID=1806527 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[INFO] Fixing CA certificate: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 08:56:10 UTC] USER=www-data EUID=0 PID=1806548 ACTION=fsop ARGS=chmod 644 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[OK]   Permissions fixed
[OK]   postgres user can now read /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt after permission fix
[2026-01-02 08:56:10 UTC] USER=www-data EUID=0 PID=1806571 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[2026-01-02 08:56:10 UTC] USER=www-data EUID=0 PID=1806594 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[2026-01-02 08:56:10 UTC] USER=www-data EUID=0 PID=1806615 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[INFO] Checking primary database size before pg_basebackup...
[INFO] Total primary database size: 29 MB
[INFO] Estimated transfer time: ~0 minutes (at 10MB/s with compression)
[INFO] Retrieving replicator password from AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/replicator
[OK]   Replicator password retrieved successfully
[INFO] Starting pg_basebackup...
[2026-01-02 08:56:12 UTC] USER=www-data EUID=0 PID=1806690 ACTION=passthru ARGS=sudo -u postgres env PGPASSWORD=qrzga0rZrBWHXjHNfE1t9bdwqo6QF84R PGSSLMODE=verify-full PGSSLCERT=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt PGSSLKEY=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key PGSSLROOTCERT=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /usr/lib/postgresql/17/bin/pg_basebackup -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -p 5432 -U replicator -D /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01 -Fp -Xs -P -R --checkpoint=fast --wal-method=stream --verbose
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/2000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1806699"
18368/30540 kB (60%), 0/1 tablespace (...er-01-standby-01/base/16384/2615)
30550/30550 kB (100%), 0/1 tablespace (...-01-standby-01/global/pg_control)
30550/30550 kB (100%), 1/1 tablespace                                         
pg_basebackup: write-ahead log end point: 0/2000120
pg_basebackup: waiting for background process to finish streaming ...
pg_basebackup: syncing data to disk ...
pg_basebackup: renaming backup_manifest.tmp to backup_manifest
pg_basebackup: base backup completed
[OK]   pg_basebackup complete
[INFO] Fixing postgresql.auto.conf to use IP-based primary_conninfo (matching golden backup)...
[2026-01-02 08:56:14 UTC] USER=www-data EUID=0 PID=1806710 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-02 08:56:14 UTC] USER=www-data EUID=0 PID=1806732 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-02 08:56:14 UTC] USER=www-data EUID=0 PID=1806753 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-02 08:56:15 UTC] USER=www-data EUID=0 PID=1806762 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[OK]   standby.signal verified and permissions set
[INFO] Fixing postgresql.conf with standby-specific settings...
[WARN] postgresql.conf not found at /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/postgresql.conf
[INFO] Verifying postgresql.auto.conf...
[WARN] postgresql.auto.conf not found - pg_basebackup may have failed
[2026-01-02 08:56:15 UTC] USER=www-data EUID=0 PID=1806785 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.Qv2JEO
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-02 08:56:15 UTC] USER=www-data EUID=0 PID=1806834 ACTION=fsop ARGS=cp /tmp/tmp.okctJjvdbw /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/pg_hba.conf
[2026-01-02 08:56:15 UTC] USER=www-data EUID=0 PID=1806855 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/pg_hba.conf
[2026-01-02 08:56:15 UTC] USER=www-data EUID=0 PID=1806878 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01-standby-01.service
[2026-01-02 08:56:15 UTC] USER=www-data EUID=0 PID=1806903 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.Aln692 /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01-standby-01.service
[2026-01-02 08:56:15 UTC] USER=www-data EUID=0 PID=1806924 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01-standby-01.service
[OK]   systemd unit written
[2026-01-02 08:56:15 UTC] USER=www-data EUID=0 PID=1806945 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 08:56:15 UTC] USER=www-data EUID=0 PID=1806966 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 08:56:16 UTC] USER=www-data EUID=0 PID=1806987 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-02 08:56:17 UTC] USER=www-data EUID=0 PID=1807107 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-worker-01-standby-01.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-02 08:56:18 UTC] USER=www-data EUID=0 PID=1807156 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-worker-01-standby-01.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Configuring synchronous replication on primary worker-01...
[INFO] Current synchronous_standby_names: ''
[INFO] Initializing synchronous_standby_names with first standby
[INFO] New synchronous_standby_names: 'ANY 1 (worker_01_standby_01)'
[2026-01-02 08:56:19 UTC] USER=www-data EUID=0 PID=1807225 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_commit = on;
ALTER SYSTEM
[2026-01-02 08:56:19 UTC] USER=www-data EUID=0 PID=1807248 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_standby_names = 'ANY 1 (worker_01_standby_01)';
ALTER SYSTEM
[2026-01-02 08:56:19 UTC] USER=www-data EUID=0 PID=1807308 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   ✅ Synchronous replication configured on primary
[OK]      Setting: ANY 1 (worker_01_standby_01)
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Skipping database/role provisioning on standby node (read-only)
[INFO]   Database/roles will be replicated from primary: worker-01
[INFO] Applying connection and memory optimizations...
[INFO] Standby will use primary's max_connections: 100
[INFO] Current settings: max_connections=100, work_mem=8MB
[INFO] Target settings (standby): max_connections=100, work_mem=8MB
[OK]   Connection settings already optimized
[INFO] Skipping password setting - this is a standby (read-only)
[INFO] Use primary's postgres password to connect to this standby
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] PostgreSQL IP: 10.100.1.211
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01-standby-01...
[INFO]   db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.211

[INFO]   ✅ db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.211    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com


[OK]   PostgreSQL 'identity-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key \
        host=db-identity-sau-main-dev-postgresql-worker-01-standby-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-worker-01-standby-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.211
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-worker-01-standby-01
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 8eaa8059-bede-4f71-ae1d-d26590a898da
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 📦 03 role...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 08:56:26 UTC] USER=www-data EUID=0 PID=1807932 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
⚠ This is a PostgreSQL STANDBY (read-only replica)
⚠ Skipping role creation - standby gets roles from primary via replication
⚠ Use the PRIMARY's credentials to connect to this standby


[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=worker-01-standby-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║                    IAM Database Schema Initialization                       ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] 🟢 Starting IAM schema provisioning...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

[INFO] 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: core
  Core Identity Directory (tenants, realms, identities, devices, MFA)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [1/20]: core/01-tenant
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "pgcrypto" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "citext" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
NOTICE:  schema "utils" already exists, skipping
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
NOTICE:  schema "core" already exists, skipping
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
NOTICE:  relation "tenant" already exists, skipping
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[OK] Table core/01-tenant initialized

[INFO] 🔸 Table [2/20]: core/02-realm
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  relation "realm" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_realm_keycloak_id" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_realm_tenant" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[OK] Table core/02-realm initialized

[INFO] 🔸 Table [3/20]: core/03-identity
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  relation "identity" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_unique_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_unique_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_type" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[OK] Table core/03-identity initialized

[INFO] 🔸 Table [4/20]: core/04-device
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  relation "device" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_device_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_fingerprint" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_trusted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_last_seen" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[OK] Table core/04-device initialized

[INFO] 🔸 Table [5/20]: core/05-identity_account
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  relation "identity_account" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_account_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_lockout" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_last_login" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[OK] Table core/05-identity_account initialized

[INFO] 🔸 Table [6/20]: core/06-identity_mfa
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  relation "identity_mfa" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_mfa_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_active" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[OK] Table core/06-identity_mfa initialized

[INFO] 🔸 Table [7/20]: core/07-external_idp_link
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  relation "external_idp_link" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_external_idp_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_provider" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_email" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[OK] Table core/07-external_idp_link initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: policy
  RBAC/ABAC Authorization (clients, roles, permissions, policies)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [8/20]: policy/01-client
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
NOTICE:  schema "policy" already exists, skipping
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  relation "client" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_client_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_key" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_status" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[OK] Table policy/01-client initialized

[INFO] 🔸 Table [9/20]: policy/02-resource
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  relation "resource" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_resource_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_external" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_owner" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[OK] Table policy/02-resource initialized

[INFO] 🔸 Table [10/20]: policy/03-scope
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  relation "scope" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_scope_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_scope_name" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[OK] Table policy/03-scope initialized

[INFO] 🔸 Table [11/20]: policy/04-permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  relation "permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_permission_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_resource" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[OK] Table policy/04-permission initialized

[INFO] 🔸 Table [12/20]: policy/05-role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  relation "role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_keycloak" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[OK] Table policy/05-role initialized

[INFO] 🔸 Table [13/20]: policy/06-role_permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  relation "role_permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_permission_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_permission_perm" already exists, skipping
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[OK] Table policy/06-role_permission initialized

[INFO] 🔸 Table [14/20]: policy/07-identity_role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  relation "identity_role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_role_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_active" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[OK] Table policy/07-identity_role initialized

[INFO] 🔸 Table [15/20]: policy/08-policy_rule
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  relation "policy_rule" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_policy_rule_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_enabled" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_priority" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[OK] Table policy/08-policy_rule initialized

[INFO] 🔸 Table [16/20]: policy/09-api_key
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  relation "api_key" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_api_key_prefix" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[OK] Table policy/09-api_key initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: audit
  Audit & Risk Logging (auth events, admin actions, risk decisions)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [17/20]: audit/01-auth_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
NOTICE:  schema "audit" already exists, skipping
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
NOTICE:  relation "auth_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_auth_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_result" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_ip" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_session" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_trace" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_risk" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[OK] Table audit/01-auth_event initialized

[INFO] 🔸 Table [18/20]: audit/02-admin_action
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
NOTICE:  relation "admin_action" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_admin_action_actor" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_target" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_trace" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[OK] Table audit/02-admin_action initialized

[INFO] 🔸 Table [19/20]: audit/03-risk_decision
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
NOTICE:  relation "risk_decision" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_risk_decision_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_level" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_decision" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_auth" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[OK] Table audit/03-risk_decision initialized

[INFO] 🔸 Table [20/20]: audit/04-consent_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
NOTICE:  relation "consent_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_consent_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_version" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_granted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  relation "audit.auth_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  relation "audit.auth_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  relation "audit.admin_action_2026_03" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  relation "audit.admin_action_2026_04" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  relation "audit.risk_decision_2026_03" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  relation "audit.risk_decision_2026_04" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  relation "audit.consent_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  relation "audit.consent_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[OK] Table audit/04-consent_event initialized


════════════════════════════════════════════════════════════════════════════
[OK] ✅ IAM Schema Initialization Complete!
[OK] All 20 tables initialized successfully

Schemas created:
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

Design highlights:
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

════════════════════════════════════════════════════════════════════════════

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=worker-01-standby-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: identity/run.sh started for SERVICE=identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity
🔍 DEBUG_CHECKPOINT_A3: SERVICE_ROOT exists, discovering table folders
🔍 DEBUG_CHECKPOINT_A4: Found subfolder: auth
🔍 DEBUG_CHECKPOINT_A4b: Checking for nested schema layout in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth
🔍 DEBUG_CHECKPOINT_A4c: Found nested steps dir: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps (display: auth/login)
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: auth/login|/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
[INFO] 📚 Detected grouped table folders under identity/: auth/login

🔍 DEBUG_CHECKPOINT_A7: Current IDENTIFIER=coordinator
🔍 DEBUG_CHECKPOINT_A8_PROCEED: Processing tables on coordinator/main node
🔍 DEBUG_CHECKPOINT_A9: Processing table: auth/login at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
[INFO] 🔸 Table group: auth/login
🔍 DEBUG_CHECKPOINT_A10: About to run numbered steps for table: auth/login
🔍 DEBUG_CHECKPOINT_B1: run_all_numbered_steps_in_dir called for dir=/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps table=auth/login
🔍 DEBUG_CHECKPOINT_B2: Found 1 numbered steps: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B3: About to run step: 01-init-schema.sh
Ab substep 0 compelete start
[DEBUG] Tracking substep start: steps/01-install/steps/identity/auth/login/01-init-schema (RUN_UUID=c59abb17-ebdb-4e7e-b661-4807beca42d4)
Ab substep 0 compelete start
[INFO] 📦 01 init schema...
Ab substep 1 compelete start
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing auth.login_account table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating auth schema...
NOTICE:  schema "auth" already exists, skipping
CREATE SCHEMA
✅ Schema created
🔧 Creating account_status ENUM...
DO
✅ ENUM created
🔧 Creating auth.login_account table...
NOTICE:  relation "login_account" already exists, skipping
CREATE TABLE
✅ Table created (Citus-compatible with region_hint in all constraints)
🔧 Creating indexes...
NOTICE:  relation "idx_login_account_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_login_account_username" already exists, skipping
CREATE INDEX
✅ Indexes created
ℹ️  Table already registered with Citus
🎉 Schema initialization complete for fastorder_identity_sau_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

📊 Registering environment in monitoring database (obs schema)...
   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
   Resource IP: 142.93.238.16
⚠️  Could not connect to monitoring database, skipping registration
   You can manually register later using:
   /opt/fastorder/bash/scripts/env_app_setup/setup/04-postgresql/steps/register-authN-af-aaaa1-dev.sh

==========================================
✅ Schema initialization complete!
==========================================
Ab substep 1 compelete end
Ab substep 2 compelete start
Ab substep 2 compelete end

🔍 DEBUG_CHECKPOINT_B4: Completed step: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B5: All numbered steps completed for /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A11: Completed numbered steps for table: auth/login
compeleted here

🔍 DEBUG_CHECKPOINT_A12: All tables processed
End of 04-postgresql/steps/01-install/steps/identity/run.sh

✓ ✅ Standby worker-01-standby-01 setup completed

✓ ✅ PostgreSQL installation completed
[INFO] Discovering additional setup steps...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 02-pg-bouncer.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up PgBouncer connection pooling...
[2026-01-02 08:58:03 UTC] USER=www-data EUID=0 PID=1811337 ACTION=fsop ARGS=rm -f /tmp/pgbouncer-ip.service /tmp/pgbouncer.service
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] Checking for existing PgBouncer application environment in topology …
[OK]   Using existing PgBouncer environment:
[INFO]   IP:     10.100.1.204
[INFO]   FQDN:   db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Domain: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO] Ensuring /etc/hosts entry for db-identity-sau-main-dev-postgresql-bouncer.fastorder.com …
[OK]   /etc/hosts already contains entry for db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[WARN] IP 10.100.1.204 is assigned to multiple interfaces:
    inet 10.100.1.103/32 scope global lo
       valid_lft forever preferred_lft forever
    inet 10.100.1.204/32 scope global lo:pgbouncer
--
    inet 10.100.1.214/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.100.1.204/32 scope global eth0:pgbouncer
[WARN] This may cause routing issues
[INFO] Final verification of /etc/hosts entry for db-identity-sau-main-dev-postgresql-bouncer.fastorder.com …
[OK]   /etc/hosts correctly maps db-identity-sau-main-dev-postgresql-bouncer.fastorder.com to 10.100.1.204
[OK]   PgBouncer IP 10.100.1.204 already correctly bound to lo:pgbouncer
[2026-01-02 08:58:04 UTC] USER=www-data EUID=0 PID=1811418 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 08:58:06 UTC] USER=www-data EUID=0 PID=1811515 ACTION=passthru ARGS=systemctl restart pgbouncer-ip@identity-sau-main-dev.service
[2026-01-02 08:58:06 UTC] USER=www-data EUID=0 PID=1811526 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer-ip@identity-sau-main-dev.service
[OK]   pgbouncer-ip@identity-sau-main-dev.service is active
[2026-01-02 08:58:06 UTC] USER=www-data EUID=0 PID=1811550 ACTION=fsop ARGS=mkdir -p /etc/pgbouncer/identity-sau-main-dev
[2026-01-02 08:58:06 UTC] USER=www-data EUID=0 PID=1811559 ACTION=fsop ARGS=mkdir -p /run/pgbouncer/identity-sau-main-dev
[2026-01-02 08:58:06 UTC] USER=www-data EUID=0 PID=1811568 ACTION=fsop ARGS=mkdir -p /var/log/pgbouncer/identity-sau-main-dev
[2026-01-02 08:58:06 UTC] USER=www-data EUID=0 PID=1811577 ACTION=fsop ARGS=chmod 750 /etc/pgbouncer/identity-sau-main-dev
[2026-01-02 08:58:06 UTC] USER=www-data EUID=0 PID=1811588 ACTION=fsop ARGS=chmod 750 /run/pgbouncer/identity-sau-main-dev
[2026-01-02 08:58:06 UTC] USER=www-data EUID=0 PID=1811597 ACTION=fsop ARGS=chmod 750 /var/log/pgbouncer/identity-sau-main-dev
[2026-01-02 08:58:06 UTC] USER=www-data EUID=0 PID=1811606 ACTION=fsop ARGS=chown root:postgres /etc/pgbouncer/identity-sau-main-dev
[2026-01-02 08:58:06 UTC] USER=www-data EUID=0 PID=1811615 ACTION=fsop ARGS=chown postgres:postgres /run/pgbouncer/identity-sau-main-dev
[2026-01-02 08:58:06 UTC] USER=www-data EUID=0 PID=1811624 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbouncer/identity-sau-main-dev
[INFO] Generating pgbouncer_admin client certificates...
[INFO] ⏳ This may take 30-60 seconds...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    pgbouncer_admin
Identifier:  pgbouncer
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        pgbouncer
  User (CN):   pgbouncer_admin
  Hostname:    db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:58:07 UTC] USER=www-data EUID=0 PID=1811660 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-pgbouncer-pgbouncer_admin
[2026-01-02 08:58:07 UTC] USER=www-data EUID=0 PID=1811669 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-02 08:58:07 UTC] USER=www-data EUID=0 PID=1811678 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
[2026-01-02 08:58:07 UTC] USER=www-data EUID=0 PID=1811687 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-02 08:58:07 UTC] USER=www-data EUID=0 PID=1811696 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = pgbouncer_admin
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:07 UTC] USER=www-data EUID=0 PID=1811711 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:07 UTC] USER=www-data EUID=0 PID=1811720 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811729 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811738 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811747 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811756 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811766 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811775 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811784 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811793 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811802 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811813 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811822 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811831 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811840 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811849 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811858 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811867 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811896 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811905 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811914 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811923 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811932 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811941 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811950 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 08:58:08 UTC] USER=www-data EUID=0 PID=1811959 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1811968 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1811977 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1811986 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1811996 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812006 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812018 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812029 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812038 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812047 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812056 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812065 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812074 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812083 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812094 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812104 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812114 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812124 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812133 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812142 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812151 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812160 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:09 UTC] USER=www-data EUID=0 PID=1812169 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812179 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812189 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812198 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812207 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812221 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812231 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812243 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812252 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812262 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812271 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812280 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812289 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812298 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812307 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812316 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812325 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812334 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812344 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:58:10 UTC] USER=www-data EUID=0 PID=1812354 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812363 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812372 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812381 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812390 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812399 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812408 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812417 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812427 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: pgbouncer_admin
Node: pgbouncer
FQDN: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-bouncer.fastorder.com -U pgbouncer_admin -d postgres

[OK]   mTLS client certificate present: /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[INFO] Creating symlinks to canonical certificates in /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend...
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812450 ACTION=fsop ARGS=mkdir -p /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812459 ACTION=fsop ARGS=mkdir -p /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812468 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812477 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812486 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt
[INFO] Creating coordinator CA symlink for PostgreSQL server verification...
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812495 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] Verifying canonical certificate permissions...
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812504 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812513 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812524 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812533 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[OK]   Backend certificate symlinks created in /etc/ssl
[OK]   Coordinator CA symlink created for server verification
[OK]   Certificates already in canonical location - no symlinks needed
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812562 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 08:58:11 UTC] USER=www-data EUID=0 PID=1812571 ACTION=fsop ARGS=test -r /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] PgBouncer will use PostgreSQL coordinator CA: /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[OK]   PostgreSQL coordinator at db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432 is reachable
[INFO] Dumping SCRAM secrets from coordinator for PgBouncer auth_file …
[2026-01-02 08:58:12 UTC] USER=www-data EUID=0 PID=1812592 ACTION=fsop ARGS=cp /tmp/tmp.wj7SrRktpA /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-02 08:58:12 UTC] USER=www-data EUID=0 PID=1812601 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-02 08:58:12 UTC] USER=www-data EUID=0 PID=1812610 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[OK]   Auth file written: /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[INFO] Retrieved password from vault for pgbouncer_admin
[INFO] Ensuring PgBouncer admin role 'pgbouncer_admin' exists in Postgres (coordinator) …
[OK]   Role pgbouncer_admin created/updated successfully
[SECRETS] Setting credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/pgbouncer_admin
✓ [SECRETS] Credentials updated in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/pgbouncer_admin
[INFO] ✅ PgBouncer admin password stored in centralized secrets vault
[INFO] Re-fetching SCRAM secrets after role creation to ensure pgbouncer_admin is included …
[2026-01-02 08:58:20 UTC] USER=www-data EUID=0 PID=1812787 ACTION=fsop ARGS=cp /tmp/tmp.AMgRTi0RbB /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-02 08:58:20 UTC] USER=www-data EUID=0 PID=1812796 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-02 08:58:20 UTC] USER=www-data EUID=0 PID=1812805 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[OK]   Auth file updated with pgbouncer_admin SCRAM hash
[INFO] Auth file contains [2026-01-02 08:58:20 UTC] USER=www-data EUID=0 PID=1812815 ACTION=passthru ARGS=bash -c wc -l < '/etc/pgbouncer/identity-sau-main-dev/userlist.txt'
4 user(s)
[OK]   Admin 'pgbouncer_admin' password generated and saved
[INFO] Configuring PostgreSQL to prevent Citus metadata sync hangs...
ALTER ROLE
[OK]   Disabled Citus metadata sync for pgbouncer_admin
[INFO] Verifying application database fastorder_identity_sau_main_dev_db exists...
[OK]   ✓ Database fastorder_identity_sau_main_dev_db exists
[INFO] Granting permissions to pgbouncer_admin on fastorder_identity_sau_main_dev_db...
GRANT
[OK]   ✓ Granted CONNECT on fastorder_identity_sau_main_dev_db to pgbouncer_admin
GRANT
[OK]   ✓ Granted USAGE on schema public to pgbouncer_admin
GRANT
[OK]   ✓ Granted SELECT on all tables to pgbouncer_admin
ALTER DATABASE
[OK]   Set synchronous_commit=local for fastorder_identity_sau_main_dev_db
[INFO] Ensuring pg_hba.conf entry for pgbouncer_admin …
[INFO] Adding pg_hba.conf entries for pgbouncer_admin with cert auth …
[2026-01-02 08:58:21 UTC] USER=unknown EUID=33 PID=1812854 ACTION=-u ARGS=postgres bash
ERROR: Invalid or unauthorized action: -u
[OK]   pg_hba.conf updated and PostgreSQL configuration reloaded
[WARN] pg_hba.conf entry may not have loaded correctly
[INFO] Writing /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini …
[2026-01-02 08:58:22 UTC] USER=www-data EUID=0 PID=1812883 ACTION=fsop ARGS=cp /tmp/tmp.rGb4VNOwlH /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
[2026-01-02 08:58:22 UTC] USER=www-data EUID=0 PID=1812905 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
[2026-01-02 08:58:22 UTC] USER=www-data EUID=0 PID=1812937 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
[2026-01-02 08:58:22 UTC] USER=www-data EUID=0 PID=1812965 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbouncer/identity-sau-main-dev /run/pgbouncer/identity-sau-main-dev /var/log/pgbouncer/identity-sau-main-dev
[2026-01-02 08:58:22 UTC] USER=www-data EUID=0 PID=1812995 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[OK]   pgbouncer.ini ready
[INFO] Verifying TLS settings in pgbouncer.ini:
[2026-01-02 08:58:22 UTC] USER=www-data EUID=0 PID=1813024 ACTION=fsop ARGS=grep -E (client_tls_sslmode|server_tls) /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
client_tls_sslmode = verify-full
server_tls_sslmode = verify-full
server_tls_ca_file = /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
server_tls_cert_file = /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
server_tls_key_file  = /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[INFO] Verifying PgBouncer server certificate files:
[2026-01-02 08:58:22 UTC] USER=www-data EUID=0 PID=1813035 ACTION=fsop ARGS=test -r /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[OK]   Server cert readable by postgres: /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-01-02 08:58:22 UTC] USER=www-data EUID=0 PID=1813049 ACTION=fsop ARGS=test -r /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[OK]   Server key readable by postgres: /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[INFO] Verifying coordinator CA certificate:
[2026-01-02 08:58:22 UTC] USER=www-data EUID=0 PID=1813073 ACTION=fsop ARGS=test -r /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[OK]   Coordinator CA readable by postgres: /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] Preflight: stopping any conflicting PgBouncer on 6432 …
[2026-01-02 08:58:22 UTC] USER=www-data EUID=0 PID=1813106 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer.service
[2026-01-02 08:58:22 UTC] USER=www-data EUID=0 PID=1813146 ACTION=passthru ARGS=systemctl stop pgbouncer@identity-sau-main-dev.service
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json?all=1": dial unix /var/run/docker.sock: connect: permission denied
[WARN] Killing existing pgbouncer processes: 1750126
1750275
[2026-01-02 08:58:23 UTC] USER=www-data EUID=0 PID=1813199 ACTION=passthru ARGS=bash -c kill -9 1750126
[2026-01-02 08:58:23 UTC] USER=www-data EUID=0 PID=1813210 ACTION=passthru ARGS=bash -c kill -9 1750275
[2026-01-02 08:58:25 UTC] USER=www-data EUID=0 PID=1813248 ACTION=passthru ARGS=systemctl daemon-reload
[OK]   systemd unit installed: pgbouncer@identity-sau-main-dev.service
[INFO] Running pre-flight IP conflict check for 10.100.1.204:6432 …
[WARN] IP conflict checker not found at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/check-ip-conflicts.sh
[WARN] Skipping pre-flight check - conflicts may occur
[INFO] Starting PgBouncer (identity-sau-main-dev) …
[2026-01-02 08:58:26 UTC] USER=www-data EUID=0 PID=1813346 ACTION=passthru ARGS=systemctl restart pgbouncer@identity-sau-main-dev.service
[2026-01-02 08:58:26 UTC] USER=www-data EUID=0 PID=1813357 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer@identity-sau-main-dev.service
[OK]   Service ACTIVE
[INFO] Verifying auth_file before probing …
[INFO] Auth file contains 4 user(s)
[WARN] Auth file does NOT contain pgbouncer_admin entry - authentication will fail
[INFO] Probing admin console via SSL (psql to database 'pgbouncer') …
[INFO] Retrieved password from vault for admin console probe
[WARN] SSL connection issue detected
[INFO] Attempting connection with sslmode=disable for testing...
[WARN] If this fails, check PgBouncer client_tls_sslmode setting
[WARN] Admin console probe failed (see error below)
psql: error: connection to server at "10.100.1.204", port 6432 failed: SSL error: certificate verify failed
[WARN] Troubleshooting:
[WARN]   1. Check auth_file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[WARN]   2. Test with: PGPASSWORD='kppzNMG6WDrJWGUYcBARr4ME' psql -h 10.100.1.204 -p 6432 -U pgbouncer_admin -d pgbouncer
[WARN]   3. Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@identity-sau-main-dev.service -n 50

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Running Comprehensive PgBouncer Verification Tests
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Password extracted: kppzNMG6WD... (using postgres user certificates)

[INFO] Test 1/7: Admin Console - SHOW POOLS
[WARN] ✗ SHOW POOLS: FAILED
[WARN] Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@identity-sau-main-dev.service -n 50

[INFO] Test 2/7: Admin Console - SHOW VERSION
[WARN] ✗ SHOW VERSION: FAILED

[INFO] Test 3/7: Admin Console - SHOW STATS
[WARN] ✗ SHOW STATS: FAILED

[INFO] Test 4/7: Admin Console - SHOW DATABASES
[WARN] ✗ SHOW DATABASES: FAILED

[INFO] Test 5/7: Admin Console - SHOW CONFIG
[WARN] ✗ SHOW CONFIG: FAILED
psql   "host=db-identity-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_identity_sau_main_dev_db user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME    connect_timeout=5 sslmode=verify-full    sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt    sslcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt    sslkey=/etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key"   --no-psqlrc -Atc 'SELECT version();'

[INFO] Test 6/7: Application Database - SELECT version()
[WARN] ✗ Application database query: FAILED (timeout or connection issue)
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[INFO] Test 7/8: Application Database - Connection Details
[WARN] ✗ Connection details: FAILED (timeout or connection issue)
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[INFO] Test 8/8: End-to-End Application Routing - Pool Verification
[INFO]   Running actual queries through PgBouncer to verify routing and pooling...
[WARN] ✗ End-to-end routing verification: FAILED - All 3 queries failed
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[WARN]    Otherwise check if database fastorder_identity_sau_main_dev_db exists and user pgbouncer_admin has permissions

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Verification Complete - Tests 1-5 PASSED (Admin console verified)
[WARN]   Tests 6-8 FAILED - Application database not accessible
[WARN]   This is expected if Citus is not set up yet
[WARN]   Run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[OK]   PgBouncer is up for identity-sau-main-dev

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Connection Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Password stored in: AWS Secrets Manager (fastorder/db/web/ksa/main/dev/postgresqlidentity/sau/main/dev/coordinator-pgbouncer_admin)
Current password: kppzNMG6WDrJWGUYcBARr4ME

1. Admin Console (using IP address to avoid DNS/SSL issues):
   psql "host=10.100.1.204 port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME sslmode=verify-full sslrootcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

2. Admin Console (using hostname):
   psql "host=db-identity-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME sslmode=verify-full sslrootcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

3. Application Database:
   psql "host=db-identity-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_identity_sau_main_dev_db sslkey=/etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME sslmode=verify-full sslrootcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

4. Using .pgpass file:
   echo "db-identity-sau-main-dev-postgresql-bouncer.fastorder.com:6432:*:pgbouncer_admin:kppzNMG6WDrJWGUYcBARr4ME" >> ~/.pgpass
   chmod 600 ~/.pgpass
   psql -h db-identity-sau-main-dev-postgresql-bouncer.fastorder.com -p 6432 -U pgbouncer_admin -d fastorder_identity_sau_main_dev_db

5. Retrieve password from vault:
   source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
   PGPASSWORD="$(get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password')" \
     psql -h 10.100.1.204 -p 6432 -U pgbouncer_admin -d pgbouncer -c "SHOW POOLS;"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Architecture
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  • Default db 'fastorder_identity_sau_main_dev_db' → Citus coordinator (db-identity-sau-main-dev-postgresql-coordinator.fastorder.com)
  • Worker access: 'fastorder_identity_sau_main_dev_db_worker_1', 'fastorder_identity_sau_main_dev_db_worker_2', … (if exist)
  • Client TLS: require (password auth) / verify-full (mTLS with certs)
  • Server TLS: verify-full (PgBouncer validates PostgreSQL certs)
  • Auth: SCRAM-SHA-256 via /etc/pgbouncer/identity-sau-main-dev/userlist.txt
  • Pool mode: transaction (stateless connections)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Management
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Service Status:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer@identity-sau-main-dev.service
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@identity-sau-main-dev.service

Logs:
  command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@identity-sau-main-dev.service -f
  /usr/local/bin/fastorder-provisioning-wrapper.sh tail -f /var/log/pgbouncer/identity-sau-main-dev/pgbouncer.log

Reload Config:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@identity-sau-main-dev.service

Restart:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@identity-sau-main-dev.service

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Files
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Config:        /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
Auth file:     /etc/pgbouncer/identity-sau-main-dev/userlist.txt
Server cert:   /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.crt
Server key:    /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.key
CA cert:       /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt
PG CA:         /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
Logs:          /var/log/pgbouncer/identity-sau-main-dev/pgbouncer.log

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Troubleshooting
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


If "SASL authentication failed":
  1. Check auth file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/identity-sau-main-dev/userlist.txt
  2. Verify pgbouncer_admin is present with SCRAM hash
  3. Get password from vault:
     source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
     get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password'
  4. Reload PgBouncer: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@identity-sau-main-dev.service

If "no pg_hba.conf entry":
  1. Check pg_hba.conf on coordinator
  2. Add rule: hostssl all pgbouncer_admin 10.100.1.204/32 cert clientcert=verify-full
  3. Reload PostgreSQL

To add users to PgBouncer:
  1. Create user in PostgreSQL with password
  2. Re-run SCRAM dump:
     psql "host=db-identity-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 dbname=postgres user=postgres \
       sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt \
       sslcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt sslkey=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key" \
       -Atc "SELECT '\"' || rolname || '\" \"' || rolpassword || '\"' \
             FROM pg_authid WHERE rolpassword LIKE 'SCRAM-SHA-256%' \
             AND rolcanlogin ORDER BY rolname;" | command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop tee /etc/pgbouncer/identity-sau-main-dev/userlist.txt
  3. Reload: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@identity-sau-main-dev.service

[INFO] Registering PgBouncer node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        identity-sau-main-dev-pgbouncer
[INFO]   Identifier Parent: postgresql
[INFO]   IP:                10.100.1.204
[INFO]   Port:              6432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 426480a5-2f64-4fc0-b2b5-710f9ccb059a
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   PgBouncer node registered to observability API
✓ ✅ PgBouncer setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 03-citus-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] CITUS DISTRIBUTED CLUSTER SETUP
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Phase 1: Installing Citus extension on workers...
[INFO] Phase 2: Setting up coordinator and registering workers...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 📦 PHASE 1: Installing Citus extension on 1 worker(s)...

[INFO] → Worker 1/1: Installing Citus on worker-01...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS CLUSTER SETUP
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔧 Setting up Citus Worker...
[INFO] Temporarily disabling synchronous replication for extension installation...
t
[INFO] Installing Citus extension on worker...
[OK]   Citus extension installed on worker
[INFO] Restoring synchronous replication settings...
t
[INFO] Worker Citus extension installed - registration will happen when coordinator setup runs

[OK]   Citus setup complete for worker-01
[INFO] ═══════════════════════════════════════════════════════════════════════════════
✓   ✅ Citus extension installed on worker-01

✓ ✅ Phase 1 Complete: All 1 workers have Citus extension installed

[INFO] 🔧 PHASE 2: Setting up Citus coordinator and registering workers...

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS CLUSTER SETUP
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔧 Setting up Citus Coordinator...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] DIAGNOSTIC: Configuration Variables
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PG_WORKERS_NUM: 1
[INFO] ENV_ID: identity-sau-main-dev
[INFO] DOMAIN: fastorder.com
[INFO] PORT: 5432
[INFO] SOCKET_DIR: /var/run/postgresql-identity-sau-main-dev-coordinator
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Ensuring postgres client certificates exist for coordinator...
[OK]   Postgres client certificates already exist for coordinator
[INFO] Adding citus_cert_map to coordinator pg_ident.conf...
[OK]   pg_ident.conf updated for coordinator
[INFO] Installing Citus extension on coordinator...
[OK]   Citus extension installed on coordinator (postgres database)
[INFO] Installing Citus extension on application database: fastorder_identity_sau_main_dev_db...
[OK]   Citus extension installed on application database: fastorder_identity_sau_main_dev_db
[INFO] Configuring Citus SSL connection parameters...
[2026-01-02 08:58:42 UTC] USER=www-data EUID=0 PID=1813743 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
[OK]   ✅ Citus SSL connection parameters configured: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[WARN] Node not identified as coordinator, initializing...
[INFO] Checking coordinator configuration...
[INFO] Persisting citus.local_hostname to postgresql.conf...
[2026-01-02 08:58:44 UTC] USER=www-data EUID=0 PID=1813788 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /var/lib/postgresql/17/identity-sau-main-dev/coordinator/postgresql.conf
[2026-01-02 08:58:44 UTC] USER=www-data EUID=0 PID=1813811 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
[OK]   ✅ citus.local_hostname persisted to config and reloaded
[INFO] Configuring coordinator hostname in postgres database: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432

[OK]   ✅ Coordinator hostname set to db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432 in postgres database
[INFO] Checking coordinator configuration in application database: fastorder_identity_sau_main_dev_db...
[WARN] ⚠️  Coordinator registered as 'localhost' in application database, fixing...
[INFO] Configuring coordinator hostname in application database: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432
[OK]   ✅ Coordinator hostname set to db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432 in application database
[INFO] Validating coordinator configuration before worker registration...
[OK]   ✅ Coordinator hostname validated: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[OK]   ✅ citus_tables view is accessible
[INFO] Checking coordinator self-registration...
[OK]   ✅ Coordinator is already self-registered
[INFO] Configuring coordinator shard placement policy...
[OK]   ✅ Coordinator already configured in postgres database (shouldhaveshards = false)
[WARN] ⚠️  Coordinator has 17 shards in fastorder_identity_sau_main_dev_db - cannot set shouldhaveshards=false
[WARN]    You must rebalance shards to workers first, then run this setup again
[WARN]    Skipping shouldhaveshards configuration for application database
[INFO] Registering 1 worker(s) to Citus cluster...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PRE-FLIGHT: Checking worker availability...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking worker worker-01...
[INFO]   FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[OK]   ✅ Worker worker-01 is reachable via SSL
[OK]   All workers are reachable - proceeding with registration

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Adding Citus worker: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Adding citus_cert_map to worker-01 pg_ident.conf...
[OK]   pg_ident.conf updated for worker-01
[INFO] Configuring worker worker-01 HBA for coordinator (10.100.1.213) access...
[OK]   Worker worker-01 HBA configured for coordinator (10.100.1.213)
[INFO] Adding replication rules for 3 standby(s)...
[OK]   Replication rules already exist for worker-01
[INFO] Reloading worker worker-01 to apply HBA changes...
[2026-01-02 08:58:48 UTC] USER=www-data EUID=0 PID=1813945 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
[INFO] Configuring coordinator HBA for worker worker-01 (10.100.1.214) access...
[OK]   Coordinator HBA configured for worker worker-01 (10.100.1.214)
[INFO] Reloading coordinator to apply HBA changes...
[2026-01-02 08:58:48 UTC] USER=www-data EUID=0 PID=1813978 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
[INFO] Ensuring postgres client certificates exist for worker-01...
[OK]   Postgres client certificates already exist for worker-01
[INFO] Configuring citus.node_conninfo on worker-01...
[2026-01-02 08:58:49 UTC] USER=www-data EUID=0 PID=1813994 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
[OK]   citus.node_conninfo configured on worker-01
[INFO] Temporarily relaxing sync-rep on worker worker-01...
t
[OK]   Worker worker-01 sync-rep relaxed (was: sync_commit=on)
[INFO] Ensuring Citus extension on worker databases...
CREATE EXTENSION
CREATE EXTENSION
[INFO] Running citus_add_node with 180s timeout...
NOTICE:  shards are still on the coordinator after adding the new node
HINT:  Use SELECT rebalance_table_shards(); to balance shards data between workers and coordinator or SELECT citus_drain_node('db-identity-sau-main-dev-postgresql-coordinator.fastorder.com',5432); to permanently move shards away from the coordinator.
2
[INFO] Restoring worker worker-01 sync-rep settings...
t
[OK]   Worker worker-01 sync-rep restored
[OK]   ✅ Worker db-identity-sau-main-dev-postgresql-worker-01.fastorder.com successfully added to Citus cluster
[INFO]    Node ID: 2
[INFO]    Registered in: postgres, fastorder_identity_sau_main_dev_db
[OK]   Worker worker-01 registration successful
[INFO] Configuring worker worker-01 shard placement policy...
[OK]   ✅ Worker worker-01 configured to hold shards in all databases


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] POST-REGISTRATION: Verifying cluster state...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Expected workers: 1
[INFO] Registered workers: 1
[OK]   ✅ All 1 workers successfully registered!

[INFO] Citus cluster configuration:
db-identity-sau-main-dev-postgresql-coordinator.fastorder.com  5432  0  t  primary  f
db-identity-sau-main-dev-postgresql-worker-01.fastorder.com    5432  1  t  primary  t

[INFO] Note: groupid=0 is the coordinator, groupid>0 are workers
[INFO]       shouldhaveshards: false=query router only, true=holds data shards

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] FINAL VALIDATION: Verifying configuration persistence...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:58:53 UTC] USER=www-data EUID=0 PID=1814167 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /var/lib/postgresql/17/identity-sau-main-dev/coordinator/postgresql.conf
[OK]   ✅ citus.local_hostname persisted in postgresql.conf
[OK]   ✅ All 1 worker(s) successfully registered and verified

[OK]   ✅ All validation checks passed
[OK]   Citus coordinator setup complete

[OK]   Citus setup complete for coordinator
[INFO] ═══════════════════════════════════════════════════════════════════════════════

✓ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ CITUS CLUSTER SETUP COMPLETED SUCCESSFULLY
✓    Coordinator: Ready and accepting connections
✓    Workers registered: 1
✓ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 05-backup-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up coordinator backup...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for identity-sau-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-01-02 08:58:55 UTC] USER=www-data EUID=0 PID=1814236 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-02 08:58:55 UTC] USER=www-data EUID=0 PID=1814245 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/identity-sau-main-dev
[2026-01-02 08:58:55 UTC] USER=www-data EUID=0 PID=1814254 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-01-02 08:58:55 UTC] USER=www-data EUID=0 PID=1814263 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-02 08:58:55 UTC] USER=www-data EUID=0 PID=1814272 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-02 08:58:55 UTC] USER=www-data EUID=0 PID=1814281 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-02 08:59:04 UTC] USER=www-data EUID=0 PID=1814337 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-02 08:59:04 UTC] USER=www-data EUID=0 PID=1814346 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-02 08:59:05 UTC] USER=www-data EUID=0 PID=1814355 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-01-02 08:59:05 UTC] USER=www-data EUID=0 PID=1814364 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-02 08:59:05 UTC] USER=www-data EUID=0 PID=1814373 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-identity-sau-main-dev
[2026-01-02 08:59:05 UTC] USER=www-data EUID=0 PID=1814394 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-02 08:59:05 UTC] USER=www-data EUID=0 PID=1814403 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-01-02 08:59:05 UTC] USER=www-data EUID=0 PID=1814412 ACTION=fsop ARGS=find /var/lib/postgresql/17/identity-sau-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-01-02 08:59:05 UTC] USER=www-data EUID=0 PID=1814421 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-02 08:59:05 UTC] USER=www-data EUID=0 PID=1814430 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-01-02 08:59:05 UTC] USER=www-data EUID=0 PID=1814439 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-01-02 08:59:05 UTC] USER=www-data EUID=0 PID=1814448 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-02 08:59:05 UTC] USER=www-data EUID=0 PID=1814457 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/PG_VERSION
[2026-01-02 08:59:05 UTC] USER=www-data EUID=0 PID=1814468 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza identity-sau-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-02 08:59:06 UTC] USER=www-data EUID=0 PID=1814525 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-02 08:59:08 UTC] USER=www-data EUID=0 PID=1814545 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 08:59:12 UTC] USER=www-data EUID=0 PID=1814591 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-02 08:59:12 UTC] USER=www-data EUID=0 PID=1814617 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator --log-level-console=info check
2026-01-02 08:59:12.642 P00   INFO: check command begin 2.56.0: --exec-id=1814624-6f386d3e --log-level-console=info --log-level-file=debug --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-02 08:59:12.683 P00   INFO: check repo1 configuration (primary)
2026-01-02 08:59:12.700 P00  ERROR: [028]: backup and archive info files exist but do not match the database
                                    HINT: is this the correct stanza?
                                    HINT: did an error occur during stanza-upgrade?
2026-01-02 08:59:12.700 P00   INFO: check command end: aborted with exception [028]
[WARN] ⚠️  Stanza verification failed - this may be normal if WAL archiving hasn't started yet
[WARN]    The backup system is configured and will work once WAL segments are generated

[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-02 08:59:12 UTC] USER=www-data EUID=0 PID=1814638 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-02 08:59:12 UTC] USER=www-data EUID=0 PID=1814647 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-02 08:59:12 UTC] USER=www-data EUID=0 PID=1814665 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[2026-01-02 08:59:12 UTC] USER=www-data EUID=0 PID=1814674 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-02 08:59:12 UTC] USER=www-data EUID=0 PID=1814692 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-identity-sau-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-01-02 08:59:13 UTC] USER=www-data EUID=0 PID=1814710 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 08:59:13 UTC] USER=www-data EUID=0 PID=1814720 ACTION=fsop ARGS=sed -i s|__ENV_ID__|identity-sau-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 08:59:13 UTC] USER=www-data EUID=0 PID=1814729 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/var/lib/postgresql/17/identity-sau-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 08:59:13 UTC] USER=www-data EUID=0 PID=1814738 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 08:59:13 UTC] USER=www-data EUID=0 PID=1814748 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-02 08:59:13.402 P00   INFO: start command begin 2.56.0: --exec-id=1814769-afc34023 --log-level-console=info --log-level-file=debug --stanza=identity-sau-main-dev-coordinator
2026-01-02 08:59:13.402 P00   WARN: stop file does not exist for stanza identity-sau-main-dev-coordinator
2026-01-02 08:59:13.402 P00   INFO: start command end: completed successfully (7ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-02 08:59:13.480 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=1814782-01c7e0f0 --log-level-console=info --log-level-file=debug --no-online --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-02 08:59:13.496 P00   INFO: stanza-upgrade for stanza 'identity-sau-main-dev-coordinator' on repo1
2026-01-02 08:59:13.528 P00   INFO: stanza-upgrade command end: completed successfully (52ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-02 08:59:13 UTC] USER=www-data EUID=0 PID=1814788 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260102-085913.log
[2026-01-02 08:59:13 UTC] USER=www-data EUID=0 PID=1814797 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260102-085913.log
[2026-01-02 08:59:13 UTC] USER=www-data EUID=0 PID=1814806 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260102-085913.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-01-02 08:59:24 UTC] USER=www-data EUID=0 PID=1815198 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-1814206.log /var/log/pgbackrest/initial-backup-20260102-085913.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260102-085913.log
   2026-01-02 08:59:24.791 P00   INFO: repo1: remove expired backup 20251205-100802F
   2026-01-02 08:59:24.863 P00   INFO: repo1: 17-23 remove archive, start = 000000010000000000000003, stop = 000000010000000000000005
   2026-01-02 08:59:24.864 P00   INFO: repo1: 17-24 no archive to remove
   2026-01-02 08:59:24.865 P00   INFO: repo1: 17-25 remove archive, start = 000000010000000000000003, stop = 000000010000000000000003
   2026-01-02 08:59:24.865 P00   INFO: expire command end: completed successfully (87ms)

[INFO] Current backups:
stanza: identity-sau-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000006/000000010000000000000010

        full backup: 20251205-100826F
            timestamp start/stop: 2025-12-05 10:08:26+00 / 2025-12-05 10:08:29+00
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 33.6MB, database backup size: 33.6MB
            repo1: backup set size: 5.4MB, backup size: 5.4MB

    db (prior)
        wal archive min/max (17): 000000010000000000000004/00000001000000000000000B

        full backup: 20260102-082153F
            timestamp start/stop: 2026-01-02 08:21:53+00 / 2026-01-02 08:22:04+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

        full backup: 20260102-082225F
            timestamp start/stop: 2026-01-02 08:22:25+00 / 2026-01-02 08:22:32+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

    db (current)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000004

        full backup: 20260102-085913F
            timestamp start/stop: 2026-01-02 08:59:13+00 / 2026-01-02 08:59:24+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.3MB, database backup size: 37.3MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         identity-sau-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/identity-sau-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up worker backups for 1 worker(s)...
[INFO] Setting up backup for: worker-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for identity-sau-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-01-02 08:59:25 UTC] USER=www-data EUID=0 PID=1815260 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-02 08:59:25 UTC] USER=www-data EUID=0 PID=1815269 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/identity-sau-main-dev
[2026-01-02 08:59:25 UTC] USER=www-data EUID=0 PID=1815280 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-01-02 08:59:25 UTC] USER=www-data EUID=0 PID=1815289 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-02 08:59:25 UTC] USER=www-data EUID=0 PID=1815298 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-02 08:59:25 UTC] USER=www-data EUID=0 PID=1815307 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-02 08:59:34 UTC] USER=www-data EUID=0 PID=1815385 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-02 08:59:34 UTC] USER=www-data EUID=0 PID=1815394 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-02 08:59:34 UTC] USER=www-data EUID=0 PID=1815403 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-01-02 08:59:34 UTC] USER=www-data EUID=0 PID=1815412 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-02 08:59:34 UTC] USER=www-data EUID=0 PID=1815421 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-identity-sau-main-dev
[2026-01-02 08:59:34 UTC] USER=www-data EUID=0 PID=1815442 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-02 08:59:34 UTC] USER=www-data EUID=0 PID=1815451 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-01-02 08:59:34 UTC] USER=www-data EUID=0 PID=1815461 ACTION=fsop ARGS=find /var/lib/postgresql/17/identity-sau-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-01-02 08:59:34 UTC] USER=www-data EUID=0 PID=1815470 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-02 08:59:34 UTC] USER=www-data EUID=0 PID=1815479 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-01-02 08:59:34 UTC] USER=www-data EUID=0 PID=1815488 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-01-02 08:59:34 UTC] USER=www-data EUID=0 PID=1815497 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-02 08:59:35 UTC] USER=www-data EUID=0 PID=1815506 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/PG_VERSION
[2026-01-02 08:59:35 UTC] USER=www-data EUID=0 PID=1815516 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza identity-sau-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-02 08:59:36 UTC] USER=www-data EUID=0 PID=1815589 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-02 08:59:38 UTC] USER=www-data EUID=0 PID=1815608 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 08:59:42 UTC] USER=www-data EUID=0 PID=1815651 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-02 08:59:42 UTC] USER=www-data EUID=0 PID=1815675 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator --log-level-console=info check
2026-01-02 08:59:42.758 P00   INFO: check command begin 2.56.0: --exec-id=1815685-60555da1 --log-level-console=info --log-level-file=debug --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-02 08:59:42.804 P00   INFO: check repo1 configuration (primary)
2026-01-02 08:59:42.875 P00   INFO: check repo1 archive for WAL (primary)
2026-01-02 08:59:43.176 P00   INFO: WAL segment 000000010000000000000006 successfully archived to '/var/lib/pgbackrest/backup/identity-sau-main-dev/archive/identity-sau-main-dev-coordinator/17-25/0000000100000000/000000010000000000000006-aa537dc01099eba463cf743eb85ca3bb3760da5f.lz4' on repo1
2026-01-02 08:59:43.176 P00   INFO: check command end: completed successfully (424ms)
[INFO] ✅ Stanza verification passed

[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-02 08:59:43 UTC] USER=www-data EUID=0 PID=1815711 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-02 08:59:43 UTC] USER=www-data EUID=0 PID=1815720 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-02 08:59:43 UTC] USER=www-data EUID=0 PID=1815741 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[2026-01-02 08:59:43 UTC] USER=www-data EUID=0 PID=1815751 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-02 08:59:43 UTC] USER=www-data EUID=0 PID=1815769 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-identity-sau-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-01-02 08:59:43 UTC] USER=www-data EUID=0 PID=1815787 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 08:59:43 UTC] USER=www-data EUID=0 PID=1815796 ACTION=fsop ARGS=sed -i s|__ENV_ID__|identity-sau-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 08:59:43 UTC] USER=www-data EUID=0 PID=1815805 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/var/lib/postgresql/17/identity-sau-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 08:59:43 UTC] USER=www-data EUID=0 PID=1815814 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 08:59:43 UTC] USER=www-data EUID=0 PID=1815823 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-02 08:59:43.893 P00   INFO: start command begin 2.56.0: --exec-id=1815844-2c888e10 --log-level-console=info --log-level-file=debug --stanza=identity-sau-main-dev-coordinator
2026-01-02 08:59:43.894 P00   WARN: stop file does not exist for stanza identity-sau-main-dev-coordinator
2026-01-02 08:59:43.894 P00   INFO: start command end: completed successfully (6ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-02 08:59:43.961 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=1815855-7895449f --log-level-console=info --log-level-file=debug --no-online --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-02 08:59:43.962 P00   INFO: stanza-upgrade for stanza 'identity-sau-main-dev-coordinator' on repo1
2026-01-02 08:59:43.964 P00   INFO: stanza 'identity-sau-main-dev-coordinator' on repo1 is already up to date
2026-01-02 08:59:43.964 P00   INFO: stanza-upgrade command end: completed successfully (9ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-02 08:59:44 UTC] USER=www-data EUID=0 PID=1815860 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260102-085943.log
[2026-01-02 08:59:44 UTC] USER=www-data EUID=0 PID=1815870 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260102-085943.log
[2026-01-02 08:59:44 UTC] USER=www-data EUID=0 PID=1815880 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260102-085943.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-01-02 08:59:49 UTC] USER=www-data EUID=0 PID=1815941 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-1815226.log /var/log/pgbackrest/initial-backup-20260102-085943.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260102-085943.log
   2026-01-02 08:59:49.701 P00   INFO: repo1: remove expired backup 20251205-100826F
   2026-01-02 08:59:49.755 P00   INFO: repo1: remove archive path /var/lib/pgbackrest/backup/identity-sau-main-dev/archive/identity-sau-main-dev-coordinator/17-23
   2026-01-02 08:59:49.760 P00   INFO: repo1: 17-24 no archive to remove
   2026-01-02 08:59:49.764 P00   INFO: repo1: 17-25 no archive to remove
   2026-01-02 08:59:49.765 P00   INFO: expire command end: completed successfully (82ms)

[INFO] Current backups:
stanza: identity-sau-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000004/00000001000000000000000B

        full backup: 20260102-082153F
            timestamp start/stop: 2026-01-02 08:21:53+00 / 2026-01-02 08:22:04+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

        full backup: 20260102-082225F
            timestamp start/stop: 2026-01-02 08:22:25+00 / 2026-01-02 08:22:32+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

    db (current)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000007

        full backup: 20260102-085913F
            timestamp start/stop: 2026-01-02 08:59:13+00 / 2026-01-02 08:59:24+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.3MB, database backup size: 37.3MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

        full backup: 20260102-085944F
            timestamp start/stop: 2026-01-02 08:59:44+00 / 2026-01-02 08:59:49+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.3MB, database backup size: 37.3MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         identity-sau-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/identity-sau-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ Backup setup completed for coordinator and all workers

[INFO] Skipping 06-distribute-tables-canary.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 07-distribute-tables.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 08:59:51 UTC] USER=unknown EUID=33 PID=1815996 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 08:59:51 UTC] USER=unknown EUID=33 PID=1816003 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 08:59:51 UTC] USER=unknown EUID=33 PID=1816010 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 08:59:51 UTC] USER=unknown EUID=33 PID=1816017 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS TABLE DISTRIBUTION
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔐 Secure connection established
[INFO]    Host: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432
[INFO]    Database: fastorder_identity_sau_main_dev_db
[INFO]    SSL: verify-full (TLS 1.2+)
[INFO]    Timeouts: statement=120s, idle_tx=300s

[INFO] 🔍 Running preflight checks...
[INFO] Testing database connectivity...
[OK]   ✅ Database connection successful
[OK]   ✅ Connected to correct database: fastorder_identity_sau_main_dev_db
[INFO] Checking Citus extension in database fastorder_identity_sau_main_dev_db...
[OK]   Citus version: 13.2-1
[INFO] Checking worker registration...
[OK]   Registered workers: 1
[INFO] Worker nodes:
[INFO]                             nodename                           | nodeport | isactive | noderole 
[INFO]   -------------------------------------------------------------+----------+----------+----------
[INFO]    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com |     5432 | t        | primary
[INFO]   (1 row)
[INFO]   

[INFO] 📊 Starting table distribution...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Distributing: auth.login_account
[INFO] Description: User authentication table - distributed by region for tenant isolation
[INFO] Shard key: region_hint
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📝 Current rows: 0
[INFO] Checking constraints compatibility with Citus...
[OK]   ✅ No conflicting constraints found
[OK]   ✅ Table already distributed - skipping
[INFO]    Distribution column: region_hint
[OK]   ✅ Data integrity verified (0 rows)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[OK]   ✅ All tables distributed successfully!
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 📊 Citus Cluster Summary:

[INFO] Distributed tables:
[INFO]            table          |   type    | shard_key | shards | size  
[INFO]   ------------------------+-----------+-----------+--------+-------
[INFO]    core.tenant            | reference | <none>    |      1 | 24 kB
[INFO]    core.realm             | local     | <none>    |      1 | 40 kB
[INFO]    core.identity          | local     | <none>    |      1 | 72 kB
[INFO]    core.device            | local     | <none>    |      1 | 48 kB
[INFO]    core.identity_account  | local     | <none>    |      1 | 48 kB
[INFO]    core.identity_mfa      | local     | <none>    |      1 | 40 kB
[INFO]    core.external_idp_link | local     | <none>    |      1 | 48 kB
[INFO]    policy.client          | local     | <none>    |      1 | 56 kB
[INFO]    policy.resource        | local     | <none>    |      1 | 48 kB
[INFO]    policy.scope           | local     | <none>    |      1 | 40 kB
[INFO]    policy.permission      | local     | <none>    |      1 | 48 kB
[INFO]    policy.role            | local     | <none>    |      1 | 56 kB
[INFO]    policy.role_permission | local     | <none>    |      1 | 24 kB
[INFO]    policy.identity_role   | local     | <none>    |      1 | 40 kB
[INFO]    policy.policy_rule     | local     | <none>    |      1 | 48 kB
[INFO]    policy.api_key         | local     | <none>    |      1 | 56 kB
[INFO]    auth.login_account     | reference | <none>    |      1 | 48 kB
[INFO]   (17 rows)
[INFO]   

[INFO] Worker capacity:
[INFO]    worker | total_shards | total_size 
[INFO]   --------+--------------+------------
[INFO]   (0 rows)
[INFO]   

[OK]   Citus table distribution complete

[INFO] Skipping 08-distribute-tables-rollback.sh (rollback script - run manually only)
[INFO] Skipping 09-distribute-tables-test.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 10-setup-cdc.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] CDC PIPELINE SETUP (Debezium + Elasticsearch Sink)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Log file: /var/log/fastorder/cdc/10-setup-cdc-*.log

[INFO] Running CDC setup for identifier: coordinator
[2026-01-02 09:00:03] ==========================================
[2026-01-02 09:00:03] CDC SETUP SCRIPT STARTED
[2026-01-02 09:00:03] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260102_090003.log
[2026-01-02 09:00:03] ==========================================
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 09:00:11] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 09:00:12]   CDC Pipeline Setup (Debezium + ES Sink)
[2026-01-02 09:00:12] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 09:00:13]   Environment: identity-sau-main-dev
[2026-01-02 09:00:13]   Identifier:  coordinator
[2026-01-02 09:00:13]   Service:     identity
[2026-01-02 09:00:13] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 09:00:14] 📂 CDC_BASE_DIR exists: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc
[2026-01-02 09:00:14] Looking for service folder: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity
[2026-01-02 09:00:15] 
[2026-01-02 09:00:16] 📂 Found CDC configuration for service: identity
[2026-01-02 09:00:17] Scanning for subservice directories in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity
[2026-01-02 09:00:18] Found subservice: login, checking for steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps
[2026-01-02 09:00:18] 
[2026-01-02 09:00:18] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 09:00:18]   Setting up CDC for: identity/login
[2026-01-02 09:00:19] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 09:00:19] Found 7 step script(s) in /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps
[2026-01-02 09:00:19] 
[2026-01-02 09:00:19] 🔧 Running: 01-setup-debezium-auth-login.sh
[2026-01-02 09:00:19]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/01-setup-debezium-auth-login.sh
[2026-01-02 09:00:19]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Debezium CDC Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔍 Verifying Kafka infrastructure...
✅ db-identity-sau-main-dev-postgresql.fastorder.com resolves to 10.100.1.213
🔐 psql will use client cert for mTLS.
🔐 Retrieving credentials from secrets vault...
   Clearing cached credentials for coordinator...
✅ Credentials retrieved from secrets vault
🔐 Syncing debezium_user password in PostgreSQL...
✅ debezium_user password synchronized
🔍 Checking PostgreSQL SSL status...
✅ Server SSL is ON (verify-full + client cert).
🔧 Applying publication & grants over TLS…
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

NOTICE:  publication "cdc_pub_identity" does not exist, skipping
DROP PUBLICATION
CREATE PUBLICATION
SET
NOTICE:  Added shard table auth.login_account_102024 to publication
DO
RESET
GRANT
GRANT
GRANT
✅ Publication & grants done (including Citus shard table).
⏳ Waiting for Kafka Connect @ https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors…
[2026-01-02 09:02:06] 🔗 Waiting for Kafka Connect at: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
[2026-01-02 09:02:06] ⏳ Waiting for HTTP endpoint: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
[2026-01-02 09:02:06]    Expected codes: 200,500, timeout: 300s
[2026-01-02 09:02:06] ✅ HTTP endpoint ready: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083 (code: 200, took: 0s)
[2026-01-02 09:02:06] 🔄 Testing Connect worker readiness...
[2026-01-02 09:02:07] ✅ Kafka Connect worker ready
🧹 Cleaning up existing Debezium connector and slot (if any)...
   Step 0a: Also resetting ES Sink connector offsets (required for coordinated reset)...
   → Stopping ES Sink connector pg_identity_sau_main_dev_coordinator_es_sink...
   → Deleting ES Sink connector offsets...
   ✓ ES Sink offsets deleted successfully (HTTP 200)
   → Deleting ES Sink connector (will be recreated by 02-setup-es-sink.sh)...
   ✓ ES Sink connector cleanup complete
   Step 0b: Clearing stale Debezium connector offsets from Kafka Connect...
   → Stopping connector pg_identity_sau_main_dev_debezium_postgres...
   → Deleting connector offsets (forces fresh snapshot)...
   ✓ Connector offsets deleted successfully (HTTP 200)
   Step 1: Ensuring connector is completely removed...
   Deleting connector: pg_identity_sau_main_dev_debezium_postgres (attempt 1/10)
   ✓ Connector pg_identity_sau_main_dev_debezium_postgres does not exist (HTTP 404)
   Step 2: Waiting for replication slot to become inactive...
   ✓ Slot slot_identity_sau_main_dev does not exist (clean state)
   Step 3: Dropping replication slot...
   ✓ Slot slot_identity_sau_main_dev already dropped
   Step 4: Final verification...
✅ Cleanup complete - environment is clean for fresh CDC snapshot
🔐 Checking Debezium SSL certificate permissions...
🔍 Validating Debezium SSL certificates...
🔐 Connector will use mTLS to Postgres.
  ✓ Certificate: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt
  ✓ Key: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key
  ✓ Root CA: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
ℹ️  Skipping pre-flight connectivity test (will be validated by Kafka Connect)
📤 Upserting connector: PUT https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_identity_sau_main_dev_debezium_postgres/config
   Attempt 1/5: Sending PUT request to Kafka Connect...
   (This may take up to 60s as Connect validates the configuration)
   ✅ Success (HTTP 201)

🌐 HTTP Response: 201
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Response body:
{
  "name": "pg_identity_sau_main_dev_debezium_postgres",
  "config": {
    "name": "pg_identity_sau_main_dev_debezium_postgres",
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "plugin.name": "pgoutput",
    "database.hostname": "db-identity-sau-main-dev-postgresql.fastorder.com",
    "database.port": "5432",
    "database.dbname": "fastorder_identity_sau_main_dev_db",
    "database.user": "debezium_user",
    "database.password": "uKcQ0gsU3V7zjRUUtAgSnGwXW",
    "database.sslmode": "verify-full",
    "database.sslrootcert": "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt",
    "database.sslcert": "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt",
    "database.sslkey": "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key",
    "publication.name": "cdc_pub_identity",
    "publication.autocreate.mode": "disabled",
    "slot.name": "slot_identity_sau_main_dev",
    "topic.prefix": "identity_sau_main_dev_cdc",
    "schema.include.list": "auth",
    "table.include.list": "auth.login_account,auth.login_account_[0-9]+",
    "transforms": "unwrap,route",
    "transforms.unwrap.add.fields": "op,ts_ms",
    "transforms.unwrap.delete.handling.mode": "rewrite",
    "transforms.unwrap.drop.tombstones": "false",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
    "transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.route.regex": "^identity_sau_main_dev_cdc\\.auth\\.login_account(_[0-9]+)?$",
    "transforms.route.replacement": "identity_sau_main_dev_account_router",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",
    "snapshot.mode": "always"
  },
  "tasks": [],
  "type": "source"
}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Connector upserted.
🔄 Verifying connector task startup...
✅ Debezium connector task is RUNNING
ℹ️  Source table auth.login_account has 0 rows.
ℹ️  Snapshot will be metadata-only; offsets may stay empty until first change.
⏳ Waiting for Debezium initial snapshot to complete...
   📊 Slot status: restart_lsn=0/900D2C8, confirmed_flush_lsn=0/900D300
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (0s elapsed)
   ⏳ Snapshot in progress... (5s elapsed)
   ⏳ Snapshot in progress... (10s elapsed)
   📊 Slot status: restart_lsn=0/900D2C8, confirmed_flush_lsn=0/900D300
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (15s elapsed)
   ⏳ Snapshot in progress... (20s elapsed)
   ⏳ Snapshot in progress... (25s elapsed)
   📊 Slot status: restart_lsn=0/900D2C8, confirmed_flush_lsn=0/900D300
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (30s elapsed)
   ⏳ Snapshot in progress... (35s elapsed)
   ⏳ Snapshot in progress... (40s elapsed)
   📊 Slot status: restart_lsn=0/900D2C8, confirmed_flush_lsn=0/900D300
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (45s elapsed)
   ⏳ Snapshot in progress... (50s elapsed)
   ⏳ Snapshot in progress... (55s elapsed)
   📊 Slot status: restart_lsn=0/900D2C8, confirmed_flush_lsn=0/900D300
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (60s elapsed)
   ⏳ Snapshot in progress... (65s elapsed)
   ⏳ Snapshot in progress... (70s elapsed)
   📊 Slot status: restart_lsn=0/900D2C8, confirmed_flush_lsn=0/900D300
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (75s elapsed)
   ⏳ Snapshot in progress... (80s elapsed)
   ⏳ Snapshot in progress... (85s elapsed)
   📊 Slot status: restart_lsn=0/900D2C8, confirmed_flush_lsn=0/900D300
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (90s elapsed)
   ⏳ Snapshot in progress... (95s elapsed)
   ⏳ Snapshot in progress... (100s elapsed)
   📊 Slot status: restart_lsn=0/900D2C8, confirmed_flush_lsn=0/900D300
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (105s elapsed)
   ⏳ Snapshot in progress... (110s elapsed)
   ⏳ Snapshot in progress... (115s elapsed)

⚠️  WARNING: Snapshot wait timeout (120s) on EMPTY table.
   Offsets are still empty, but source table has 0 rows.
   Proceeding anyway – CDC health will be verified by test inserts.

✅ Debezium connector is RUNNING after snapshot
🔍 Final verification: Checking Debezium offsets are recorded...
   ℹ️  Source table auth.login_account has 0 rows
   ℹ️  Skipping offset verification (no data to snapshot)
✅ Debezium connector verified RUNNING (empty source table)
🔄 Phase 2: Updating connector to snapshot.mode=initial...
✅ Connector updated to snapshot.mode=initial (HTTP 200)
✅ Connector verified RUNNING after Phase 2 update
✅ Debezium connector configured successfully (two-phase snapshot complete)
[2026-01-02 09:04:39] ✅ Completed: 01-setup-debezium-auth-login.sh
[2026-01-02 09:04:39] 
[2026-01-02 09:04:39] 🔧 Running: 02-setup-es-sink.sh
[2026-01-02 09:04:39]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/02-setup-es-sink.sh
[2026-01-02 09:04:39]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[WARN] Master/coordinator not found, using node-01
[INFO] Using ES domain: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com
🔐 Retrieving keystore passwords from secrets manager...
[INFO] Retrieving Kafka truststore password...
✅ Retrieved passwords from remote backend
✅ Retrieved Kafka truststore password
[INFO] Retrieving Elasticsearch P12 password...
[INFO] 🔍 Checking secrets backend (provider: aws)...
✅ Retrieved passwords from remote backend
[INFO] ✅ Using existing passwords from backend
✅ Retrieved/generated Elasticsearch P12 password
✅ Keystore passwords retrieved successfully
   - Kafka truststore password: yOb0eqkA... (32 chars)
   - ES P12 password: 8siDJx7z... (32 chars)
[INFO] 🔐 Clearing cached ES credentials to ensure fresh retrieval...
[INFO] [INFO] ✅ Using ES password from centralized secrets vault (identifier: node-01)
[INFO] 🔐 Verifying Elasticsearch accepts client certificate...
[INFO] ✅ Elasticsearch accepting client certificate
[INFO] 🔐 Setting up ES client keystore using Kafka client certificate...
[INFO]    Certificate: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem (signed by Fastorder RA Root CA)
[INFO] 📋 Creating ES client P12 keystore from Kafka client certificate...
[2026-01-02 09:04:48 UTC] USER=www-data EUID=0 PID=1827179 ACTION=fsop ARGS=mv /tmp/es-client-1826991.p12 /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[2026-01-02 09:04:48 UTC] USER=www-data EUID=0 PID=1827188 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[2026-01-02 09:04:48 UTC] USER=www-data EUID=0 PID=1827197 ACTION=fsop ARGS=chmod 600 /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[INFO] ✅ Created ES client keystore: /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[INFO]    Using Kafka client cert signed by Fastorder RA Root CA
[INFO] ℹ️  Using Kafka truststore and adding ES CA certificate
[2026-01-02 09:04:48 UTC] USER=www-data EUID=0 PID=1827206 ACTION=fsop ARGS=test -f /opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks
[INFO] ✅ ES CA already in truststore
[INFO] [INFO] 🔗 Waiting for Kafka Connect at: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
[INFO] [INFO] ✅ Connect HTTP ready (code 200)
[INFO] [INFO] 🔍 Verifying Debezium connector snapshot status...
[INFO] [INFO] ℹ️  Source table auth.login_account has 0 rows.
[INFO] [INFO]    Skipping Debezium snapshot wait (metadata-only snapshot on empty table).
[INFO] [INFO] 🔌 Cleaning up existing ES Sink connector: pg_identity_sau_main_dev_coordinator_es_sink
[INFO] [INFO]    → Deleting connector...
[INFO] [INFO]    HTTP 404 (404 is fine)
[INFO] [INFO] 🔐 Validating Elasticsearch credentials...
[INFO] [INFO] ✅ ES credentials validated successfully
[INFO] [INFO] 🔧 Creating required Elasticsearch ingest pipelines: identity-embed-pipeline-001
[INFO] [INFO] ✅ Pipeline identity-embed-pipeline-001 created successfully
[INFO] [INFO] 🔧 Ensuring CDC index has no default_pipeline requirement...
[INFO] [INFO] ✅ Removed default_pipeline from index (if any)
[INFO] [INFO] 🔧 Ensuring dynamic mapping is enabled...
[INFO] [INFO] ✅ Dynamic mapping enabled for identity_sau_main_dev_account_router
[DEBUG] ES_TRUSTSTORE=/opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks
[DEBUG] ES_CLIENT_P12=/opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[DEBUG] TRUSTSTORE_PASS=yOb0eqkA...
[DEBUG] P12_PASS=8siDJx7z...
== Outgoing connector config (snippet) ==
2:  "name": "pg_identity_sau_main_dev_coordinator_es_sink",
6:  "connection.url": "https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200",
19:  "index": "identity_sau_main_dev_account_router",
[INFO] ⚠️  Skipping pre-validation - will validate on PUT...
[INFO] [INFO] ✅ Proceeding to PUT
[2026-01-02 09:04:50] [1/3] Upserting connector via PUT https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_identity_sau_main_dev_coordinator_es_sink/config
🌐 HTTP 201
✅ Connector created/updated successfully
{
  "name": "pg_identity_sau_main_dev_coordinator_es_sink",
  "config": {
    "name": "pg_identity_sau_main_dev_coordinator_es_sink",
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "tasks.max": "1",
    "topics": "identity_sau_main_dev_account_router",
    "connection.url": "https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200",
    "elastic.security.protocol": "SSL",
    "elastic.https.ssl.hostname.verification": "true",
    "elastic.https.ssl.truststore.location": "/opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks",
    "elastic.https.ssl.truststore.password": "yOb0eqkAqtj8HEWebgA7nf04YlqsLw44",
    "elastic.https.ssl.truststore.type": "JKS",
    "elastic.https.ssl.keystore.location": "/opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12",
    "elastic.https.ssl.keystore.password": "8siDJx7zdDhhu5iMMZwnhZfTaGFSgCvh",
    "elastic.https.ssl.keystore.type": "PKCS12",
    "elastic.username": "elastic",
    "elastic.password": "T+kMy0e84aGeV204NzYK",
    "connection.username": "elastic",
    "connection.password": "T+kMy0e84aGeV204NzYK",
    "index": "identity_sau_main_dev_account_router",
    "key.ignore": "true",
    "schema.ignore": "true",
    "behavior.on.null.values": "delete",
    "write.method": "upsert",
    "type.name": "_doc",
    "max.in.flight.requests": "1",
    "batch.size": "2000",
    "linger.ms": "100",
    "flush.timeout.ms": "60000",
    "max.retries": "10",
    "retry.backoff.ms": "5000",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false"
  },
  "tasks": [],
  "type": "sink"
}
{
  "pg_identity_sau_main_dev_debezium_postgres": {
    "status": {
      "name": "pg_identity_sau_main_dev_debezium_postgres",
      "connector": {
        "state": "RUNNING",
        "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
      },
      "tasks": [
        {
          "id": 0,
          "state": "RUNNING",
          "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
        }
      ],
      "type": "source"
    }
  },
  "pg_identity_sau_to_universe_main_dev_es_sink": {
    "status": {
      "name": "pg_identity_sau_to_universe_main_dev_es_sink",
      "connector": {
        "state": "RUNNING",
        "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
      },
      "tasks": [
        {
          "id": 0,
          "state": "FAILED",
          "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083",
          "trace": "org.apache.kafka.common.KafkaException: Failed to load SSL keystore /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12 of type PKCS12\n\tat org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:380)\n\tat org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:352)\n\tat org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:302)\n\tat org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:162)\n\tat org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:147)\n\tat org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:100)\n\tat io.confluent.connect.elasticsearch.ConfigCallbackHandler.sslContext(ConfigCallbackHandler.java:262)\n\tat io.confluent.connect.elasticsearch.ConfigCallbackHandler.createConnectionManager(ConfigCallbackHandler.java:172)\n\tat io.confluent.connect.elasticsearch.ConfigCallbackHandler.customizeHttpClient(ConfigCallbackHandler.java:95)\n\tat org.elasticsearch.client.RestClientBuilder.createHttpClient(RestClientBuilder.java:320)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.client.RestClientBuilder.build(RestClientBuilder.java:283)\n\tat io.confluent.connect.elasticsearch.ElasticsearchClient.<init>(ElasticsearchClient.java:144)\n\tat io.confluent.connect.elasticsearch.ElasticsearchSinkTask.start(ElasticsearchSinkTask.java:82)\n\tat io.confluent.connect.elasticsearch.ElasticsearchSinkTask.start(ElasticsearchSinkTask.java:54)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:324)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doStart(WorkerTask.java:176)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:225)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:281)\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:238)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\nCaused by: java.io.IOException: keystore password was incorrect\n\tat java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159)\n\tat java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)\n\tat java.base/java.security.KeyStore.load(KeyStore.java:1473)\n\tat org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:377)\n\t... 24 more\nCaused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.\n\t... 28 more\n"
        }
      ],
      "type": "sink"
    }
  },
  "pg_identity_sau_main_dev_coordinator_es_sink": {
    "status": {
      "name": "pg_identity_sau_main_dev_coordinator_es_sink",
      "connector": {
        "state": "RUNNING",
        "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
      },
      "tasks": [],
      "type": "sink"
    }
  }
}
[INFO] [INFO] 🔗 Creating ES alias for application compatibility...
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (0s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (5s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (10s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (15s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (20s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (25s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (30s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (35s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (40s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (45s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (50s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (55s)
[WARN] ⚠️  ES index not created within 60s, skipping alias creation

🔍 Final verification: Checking ES document count...
   PostgreSQL auth.login_account: 0 rows
ℹ️  PostgreSQL table is empty - skipping ES verification
✅ Done.
[2026-01-02 09:05:52] ✅ Completed: 02-setup-es-sink.sh
[2026-01-02 09:05:52] 
[2026-01-02 09:05:52] 🔧 Running: 03-setup-es-universe-sink.sh
[2026-01-02 09:05:52]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/03-setup-es-universe-sink.sh
[2026-01-02 09:05:52]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Universe Identity ES Sink Setup (Dual-Sink Pattern)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Source Zone:  sau
  Connector:      pg_identity_sau_to_universe_main_dev_es_sink
  Source Topic:   identity_sau_main_dev_account_router
  Universe ES:      search-identity-universe-main-dev.fastorder.com:9200
  Universe Index:   identity_universe_main_dev_account_router
  Zone Field:   zone: "sau" (added to each document)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Retrieving keystore passwords from secrets manager...
✅ Retrieved passwords from remote backend
✅ Retrieved Kafka truststore password
[INFO] 🔍 Checking secrets backend (provider: aws)...
✅ Retrieved passwords from remote backend
[INFO] ✅ Using existing passwords from backend
✅ Retrieved/generated Elasticsearch P12 password
🔐 Retrieving Universe ES password...
[INFO] [INFO] ✅ Retrieved Universe ES password from vault (identifier: node-01)
❌ missing CA file: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 09:06:00] ❌ FAILED: 03-setup-es-universe-sink.sh (exit code: 1)
[2026-01-02 09:06:00] ❌ CRITICAL: This is a required step for CDC pipeline. Aborting.

[ERROR] ❌ Database infrastructure (postgresql) setup failed with exit code: 1