📊 Provisioning Job Status

Environment: Identity Sau Main Dev on web-03

❌ Failed

⏱️ Timing Summary

🕐

Requested 2026-01-02 10:39:27 1 months ago

▶️

Started 2026-01-02 10:39:28 1 months ago

🏁

Finished 2026-01-02 11:10:23 1 months ago

⏲️

Total Duration 30 minutes

📋 Job Details

Job ID: 333bba61-dfcf-4fcb-986b-f0541277e251

Action: SETUP

Status: ❌ FAILED

Environment: identity-sau-main-dev

Resource: web-03 (Provider)

Requested By: admin

Parameters:

"{\"env\": \"dev\", \"zone\": \"sau\", \"branch\": \"main\", \"db_app\": \"postgresql\", \"service\": \"identity\", \"es_nodes\": 1, \"db_enabled\": true, \"pg_standby\": 1, \"pg_workers\": 1, \"search_app\": \"elasticsearch\", \"description\": \"\", \"iam_enabled\": false, \"worker_1_ip\": \"10.100.1.42\", \"eventbus_app\": \"kafka\", \"es_https_mode\": \"direct\", \"service_es_ip\": \"10.100.1.4\", \"worker_1_fqdn\": \"db-identity-sau-main-dev-postgresql-worker-01.fastorder.com\", \"search_enabled\": true, \"service_app_ip\": \"10.100.1.2\", \"service_obs_ip\": \"10.100.1.18\", \"service_es_fqdn\": \"search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com\", \"service_otlp_ip\": \"10.100.1.30\", \"eventbus_enabled\": true, \"service_app_fqdn\": \"app-identity-sau-main-dev.fastorder.com\", \"service_audit_ip\": \"10.100.1.32\", \"service_obs_fqdn\": \"obs-identity-sau-main-dev.fastorder.com\", \"service_tempo_ip\": \"10.100.1.28\", \"service_endpoints\": \"[{\\\"ip\\\":\\\"10.100.1.3\\\",\\\"fqdn\\\":\\\"app-identity-sau-main-dev.fastorder.com\\\",\\\"service\\\":\\\"app\\\"},{\\\"ip\\\":\\\"10.100.1.5\\\",\\\"fqdn\\\":\\\"search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com\\\",\\\"service\\\":\\\"es_coordinator\\\"},{\\\"ip\\\":\\\"10.100.1.7\\\",\\\"fqdn\\\":\\\"search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com\\\",\\\"service\\\":\\\"es_node_1\\\"},{\\\"ip\\\":\\\"10.100.1.9\\\",\\\"fqdn\\\":\\\"eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com\\\",\\\"service\\\":\\\"kafka_broker_1\\\"},{\\\"ip\\\":\\\"10.100.1.11\\\",\\\"fqdn\\\":\\\"eventbus-identity-sau-main-dev-kafka-connect.fastorder.com\\\",\\\"service\\\":\\\"kafka_connect\\\"},{\\\"ip\\\":\\\"10.100.1.13\\\",\\\"fqdn\\\":\\\"schema-identity-sau-main-dev-kafka-registry.fastorder.com\\\",\\\"service\\\":\\\"kafka_registry\\\"},{\\\"ip\\\":\\\"10.100.1.15\\\",\\\"fqdn\\\":\\\"db-identity-sau-main-dev-postgresql-coordinator.fastorder.com\\\",\\\"service\\\":\\\"pg_coordinator\\\"},{\\\"ip\\\":\\\"10.100.1.17\\\",\\\"fqdn\\\":\\\"db-identity-sau-main-dev-postgresql-bouncer.fastorder.com\\\",\\\"service\\\":\\\"pgbouncer\\\"},{\\\"ip\\\":\\\"10.100.1.19\\\",\\\"fqdn\\\":\\\"obs-identity-sau-main-dev.fastorder.com\\\",\\\"service\\\":\\\"obs\\\"},{\\\"ip\\\":\\\"10.100.1.21\\\",\\\"fqdn\\\":\\\"metrics-identity-sau-main-dev-prometheus.fastorder.com\\\",\\\"service\\\":\\\"metrics\\\"},{\\\"ip\\\":\\\"10.100.1.23\\\",\\\"fqdn\\\":\\\"dashboards-identity-sau-main-dev-grafana.fastorder.com\\\",\\\"service\\\":\\\"dashboards\\\"},{\\\"ip\\\":\\\"10.100.1.25\\\",\\\"fqdn\\\":\\\"alerts-identity-sau-main-dev-alertmanager.fastorder.com\\\",\\\"service\\\":\\\"alerts\\\"},{\\\"ip\\\":\\\"10.100.1.27\\\",\\\"fqdn\\\":\\\"logstore-identity-sau-main-dev-clickhouse.fastorder.com\\\",\\\"service\\\":\\\"logs\\\"},{\\\"ip\\\":\\\"10.100.1.29\\\",\\\"fqdn\\\":\\\"traces-identity-sau-main-dev-tempo.fastorder.com\\\",\\\"service\\\":\\\"traces\\\"},{\\\"ip\\\":\\\"10.100.1.31\\\",\\\"fqdn\\\":\\\"telemetry-identity-sau-main-dev-opentelemetry.fastorder.com\\\",\\\"service\\\":\\\"telemetry\\\"},{\\\"ip\\\":\\\"10.100.1.33\\\",\\\"fqdn\\\":\\\"audit-identity-sau-main-dev.fastorder.com\\\",\\\"service\\\":\\\"audit\\\"},{\\\"ip\\\":\\\"10.100.1.35\\\",\\\"fqdn\\\":\\\"backup-identity-sau-main-dev-db-postgresql.fastorder.com\\\",\\\"service\\\":\\\"backup_pg\\\"},{\\\"ip\\\":\\\"10.100.1.37\\\",\\\"fqdn\\\":\\\"backup-identity-sau-main-dev-eventbus-kafka.fastorder.com\\\",\\\"service\\\":\\\"backup_kafka\\\"},{\\\"ip\\\":\\\"10.100.1.39\\\",\\\"fqdn\\\":\\\"backup-identity-sau-main-dev-search-elasticsearch.fastorder.com\\\",\\\"service\\\":\\\"backup_es\\\"},{\\\"ip\\\":\\\"10.100.1.41\\\",\\\"fqdn\\\":\\\"backup-identity-sau-main-dev-orchestrator.fastorder.com\\\",\\\"service\\\":\\\"backup_orchestrator\\\"}]\", \"service_otlp_fqdn\": \"telemetry-identity-sau-main-dev-opentelemetry.fastorder.com\", \"postgresql_enabled\": true, \"service_audit_fqdn\": \"audit-identity-sau-main-dev.fastorder.com\", \"service_grafana_ip\": \"10.100.1.22\", \"service_tempo_fqdn\": \"traces-identity-sau-main-dev-tempo.fastorder.com\", \"service_backup_es_ip\": \"10.100.1.38\", \"service_backup_pg_ip\": \"10.100.1.34\", \"service_es_node_1_ip\": \"10.100.1.6\", \"service_grafana_fqdn\": \"dashboards-identity-sau-main-dev-grafana.fastorder.com\", \"service_pgbouncer_ip\": \"10.100.1.16\", \"service_prometheus_ip\": \"10.100.1.20\", \"worker_1_standby_1_ip\": \"10.100.1.43\", \"service_backup_es_fqdn\": \"backup-identity-sau-main-dev-search-elasticsearch.fastorder.com\", \"service_backup_pg_fqdn\": \"backup-identity-sau-main-dev-db-postgresql.fastorder.com\", \"service_es_node_1_fqdn\": \"search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com\", \"service_log_backend_ip\": \"10.100.1.26\", \"service_pgbouncer_fqdn\": \"db-identity-sau-main-dev-postgresql-bouncer.fastorder.com\", \"service_alertmanager_ip\": \"10.100.1.24\", \"service_backup_kafka_ip\": \"10.100.1.36\", \"service_prometheus_fqdn\": \"metrics-identity-sau-main-dev-prometheus.fastorder.com\", \"worker_1_standby_1_fqdn\": \"db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com\", \"service_kafka_connect_ip\": \"10.100.1.10\", \"service_log_backend_fqdn\": \"logstore-identity-sau-main-dev-clickhouse.fastorder.com\", \"service_alertmanager_fqdn\": \"alerts-identity-sau-main-dev-alertmanager.fastorder.com\", \"service_backup_kafka_fqdn\": \"backup-identity-sau-main-dev-eventbus-kafka.fastorder.com\", \"service_kafka_broker_1_ip\": \"10.100.1.8\", \"service_kafka_registry_ip\": \"10.100.1.12\", \"service_pg_coordinator_ip\": \"10.100.1.14\", \"service_kafka_connect_fqdn\": \"eventbus-identity-sau-main-dev-kafka-connect.fastorder.com\", \"postgresql_run_verification\": true, \"service_kafka_broker_1_fqdn\": \"eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com\", \"service_kafka_registry_fqdn\": \"schema-identity-sau-main-dev-kafka-registry.fastorder.com\", \"service_pg_coordinator_fqdn\": \"db-identity-sau-main-dev-postgresql-coordinator.fastorder.com\", \"service_backup_orchestrator_ip\": \"10.100.1.40\", \"service_backup_orchestrator_fqdn\": \"backup-identity-sau-main-dev-orchestrator.fastorder.com\"}"

❌ Error: One or more steps failed. Check run logs for details.

⚠️ Job Failed

This job encountered an error. You can restart from the failed step.

📢 Viewing Old Job Attempt

This job has been restarted. You are viewing an older attempt. The logs and status shown below are from the latest retry.

🔄 View Latest Attempt Latest Status:

🔄 Resume & Restart Options

This job failed at one of the steps below. You can resume from where it failed to save time and avoid re-running successful steps.

💡

1 step failed

📝 Execution Steps (9)

0/9 completed 1 failed

0% (0/9 steps)

00-preflight-checks local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

00-terraform-provision local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

01-prepare-environment local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

02-iam local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

02-observability-cell local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

03-search local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

04-eventbus local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

05-db local

❌ FAILED

⏰ Started: 2026-01-02 10:39:28

🏁 Finished: 2026-01-02 11:10:23

⏱️ Duration: 30 minutes

📄 View Logs (643690 chars)

[0;34m[INFO][0m Using database engine from DB_ENGINE environment variable: postgresql
[0;34m[INFO][0m Cleaning up any existing locks...

[0;32m[1mStarting database engine: postgresql[0m
[1;33m═══════════════════════════════════════════════[0m

[0;34m[INFO][0m Loaded from topology.json: identity-sau-main-dev
[0;32m[2026-01-02 10:39:29][0m Loaded environment: identity-sau-main-dev
[0;32m[2026-01-02 10:39:29][0m Service: identity, Zone: sau, Branch: main, Env: dev
[0;32m[2026-01-02 10:39:29][0m VM IP: 142.93.238.16, Interface: eth0:16
[0;32m[2026-01-02 10:39:29][0m Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[0;32m[2026-01-02 10:39:29][0m PostgreSQL HA Nodes: 1, Citus Enabled: yes
[0;32m✓[0m Environment initialized successfully (mode: general)
[0;34m[INFO][0m Checking observability cell readiness: obs-identity-sau-main-dev
[1;32m[OK][0m   Observability cell endpoints registered for identity-sau-main-dev
[0;34m[INFO][0m Observability cell verified for identity-sau-main-dev
[0;34m[INFO][0m Monitoring will be configured after PostgreSQL deployment (step 10-monitoring-setup.sh)
[0;34m[INFO][0m Citus mode ENABLED
[0;34m[INFO][0m → Coordinator + 1 worker(s) + 1 standby node(s) per worker
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up coordinator (Citus control plane)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-02 10:39:30 UTC] USER=unknown EUID=33 PID=1924808 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 10:39:30 UTC] USER=unknown EUID=33 PID=1924818 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 10:39:30 UTC] USER=unknown EUID=33 PID=1924825 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-02 10:39:30 UTC] USER=unknown EUID=33 PID=1924832 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 10:39:30 UTC] USER=unknown EUID=33 PID=1924839 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-02 10:39:30 UTC] USER=unknown EUID=33 PID=1924846 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede JOB_UUID=333bba61-dfcf-4fcb-986b-f0541277e251

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: identity-sau-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.213
[0;34m[INFO][0m Primary hostname: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

[0;34m[INFO][0m Adding /etc/hosts entries for coordinator...
[0;34m[INFO][0m   1. db-identity-sau-main-dev-postgresql.fastorder.com → 10.100.1.213 (primary/short)
[0;34m[INFO][0m   2. db-identity-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.213 (compatibility)

[0;34m[INFO][0m   ✅ db-identity-sau-main-dev-postgresql.fastorder.com already exists with correct IP
[0;34m[INFO][0m   ✅ db-identity-sau-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;32m✅[0m   ✅ Network & DNS configuration complete
[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
  10.100.1.213    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  10.100.1.213    db-identity-sau-main-dev-postgresql.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 10:39:34 UTC] USER=www-data EUID=0 PID=1924977 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator and /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:39:34 UTC] USER=www-data EUID=0 PID=1924987 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-02 10:39:34 UTC] USER=www-data EUID=0 PID=1924997 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1924941
[2026-01-02 10:39:34 UTC] USER=www-data EUID=0 PID=1925006 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1924941/ra_root.crt
[2026-01-02 10:39:34 UTC] USER=www-data EUID=0 PID=1925015 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1924941/ra_root.key
[2026-01-02 10:39:34 UTC] USER=www-data EUID=0 PID=1925024 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1924941/ra_root.crt
[2026-01-02 10:39:34 UTC] USER=www-data EUID=0 PID=1925033 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1924941/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-02 10:39:37 UTC] USER=www-data EUID=0 PID=1925096 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1924941/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 10:39:37 UTC] USER=www-data EUID=0 PID=1925105 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1924941/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 10:39:37 UTC] USER=www-data EUID=0 PID=1925114 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-02 10:39:37 UTC] USER=www-data EUID=0 PID=1925125 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1924941/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:39:37 UTC] USER=www-data EUID=0 PID=1925135 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:39:38 UTC] USER=www-data EUID=0 PID=1925144 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:39:38 UTC] USER=www-data EUID=0 PID=1925153 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 10:39:38 UTC] USER=www-data EUID=0 PID=1925164 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 10:39:38 UTC] USER=www-data EUID=0 PID=1925173 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 10:39:38 UTC] USER=www-data EUID=0 PID=1925182 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 10:39:38 UTC] USER=www-data EUID=0 PID=1925191 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 10:39:38 UTC] USER=www-data EUID=0 PID=1925200 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:39:38 UTC] USER=www-data EUID=0 PID=1925209 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        coordinator
Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 10:39:40 UTC] USER=www-data EUID=0 PID=1925273 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-01-02 10:39:40 UTC] USER=www-data EUID=0 PID=1925282 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 10:39:40 UTC] USER=www-data EUID=0 PID=1925292 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-01-02 10:39:40 UTC] USER=www-data EUID=0 PID=1925301 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 10:39:40 UTC] USER=www-data EUID=0 PID=1925310 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:39:40 UTC] USER=www-data EUID=0 PID=1925329 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:39:40 UTC] USER=www-data EUID=0 PID=1925338 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:39:40 UTC] USER=www-data EUID=0 PID=1925350 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925359 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925368 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925377 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925395 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925404 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925413 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925422 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925431 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925440 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925451 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925461 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925470 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925496 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925505 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925514 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925525 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925534 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925543 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925552 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 10:39:41 UTC] USER=www-data EUID=0 PID=1925561 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:39:42 UTC] USER=www-data EUID=0 PID=1925570 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:39:42 UTC] USER=www-data EUID=0 PID=1925579 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 10:39:42 UTC] USER=www-data EUID=0 PID=1925588 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 10:39:42 UTC] USER=www-data EUID=0 PID=1925599 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:39:42 UTC] USER=www-data EUID=0 PID=1925610 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:39:42 UTC] USER=www-data EUID=0 PID=1925619 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:39:42 UTC] USER=www-data EUID=0 PID=1925628 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:39:42 UTC] USER=www-data EUID=0 PID=1925637 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:39:42 UTC] USER=www-data EUID=0 PID=1925646 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:39:42 UTC] USER=www-data EUID=0 PID=1925655 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 10:39:42 UTC] USER=www-data EUID=0 PID=1925664 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 10:39:43 UTC] USER=www-data EUID=0 PID=1925673 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:39:44 UTC] USER=www-data EUID=0 PID=1925682 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:39:44 UTC] USER=www-data EUID=0 PID=1925691 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 10:39:48 UTC] USER=www-data EUID=0 PID=1925702 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 10:41:21 UTC] USER=www-data EUID=0 PID=1925736 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:21 UTC] USER=www-data EUID=0 PID=1925991 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:22 UTC] USER=www-data EUID=0 PID=1926094 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:48:30 UTC] USER=www-data EUID=0 PID=1927608 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:31 UTC] USER=www-data EUID=0 PID=1927902 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 10:48:31 UTC] USER=www-data EUID=0 PID=1928044 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:48:31 UTC] USER=www-data EUID=0 PID=1928083 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 10:48:32 UTC] USER=www-data EUID=0 PID=1928129 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:32 UTC] USER=www-data EUID=0 PID=1928183 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:33 UTC] USER=www-data EUID=0 PID=1928198 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:48:33 UTC] USER=www-data EUID=0 PID=1928217 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:48:33 UTC] USER=www-data EUID=0 PID=1928273 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:34 UTC] USER=www-data EUID=0 PID=1928339 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:35 UTC] USER=www-data EUID=0 PID=1928532 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 10:48:35 UTC] USER=www-data EUID=0 PID=1928557 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:48:35 UTC] USER=www-data EUID=0 PID=1928629 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:48:35 UTC] USER=www-data EUID=0 PID=1928670 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 10:48:35 UTC] USER=www-data EUID=0 PID=1928709 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 10:48:35 UTC] USER=www-data EUID=0 PID=1928735 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:48:35 UTC] USER=www-data EUID=0 PID=1928776 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:48:35 UTC] USER=www-data EUID=0 PID=1928805 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:48:35 UTC] USER=www-data EUID=0 PID=1928838 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 10:48:35 UTC] USER=www-data EUID=0 PID=1928862 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 10:48:35 UTC] USER=www-data EUID=0 PID=1928892 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 10:48:35 UTC] USER=www-data EUID=0 PID=1928933 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:48:35 UTC] USER=www-data EUID=0 PID=1928955 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:48:35 UTC] USER=www-data EUID=0 PID=1928971 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:48:36 UTC] USER=www-data EUID=0 PID=1929011 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 10:48:39 UTC] USER=www-data EUID=0 PID=1929640 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 10:48:39 UTC] USER=www-data EUID=0 PID=1929692 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-01-02 10:48:39 UTC] USER=www-data EUID=0 PID=1929717 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 10:48:39 UTC] USER=www-data EUID=0 PID=1929735 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:48:40 UTC] USER=www-data EUID=0 PID=1929975 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:48:40 UTC] USER=www-data EUID=0 PID=1929990 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:48:40 UTC] USER=www-data EUID=0 PID=1930014 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 10:48:40 UTC] USER=www-data EUID=0 PID=1930044 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 10:48:40 UTC] USER=www-data EUID=0 PID=1930067 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:48:40 UTC] USER=www-data EUID=0 PID=1930083 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:48:41 UTC] USER=www-data EUID=0 PID=1930123 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 10:48:41 UTC] USER=www-data EUID=0 PID=1930149 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 10:48:41 UTC] USER=www-data EUID=0 PID=1930179 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 10:48:41 UTC] USER=www-data EUID=0 PID=1930203 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 10:48:41 UTC] USER=www-data EUID=0 PID=1930228 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 10:48:41 UTC] USER=www-data EUID=0 PID=1930272 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:48:41 UTC] USER=www-data EUID=0 PID=1930355 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:48:41 UTC] USER=www-data EUID=0 PID=1930414 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 10:48:41 UTC] USER=www-data EUID=0 PID=1930456 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 10:48:42 UTC] USER=www-data EUID=0 PID=1930565 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 10:48:42 UTC] USER=www-data EUID=0 PID=1930643 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:48:42 UTC] USER=www-data EUID=0 PID=1930780 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:43 UTC] USER=www-data EUID=0 PID=1931006 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:44 UTC] USER=www-data EUID=0 PID=1931035 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:48:44 UTC] USER=www-data EUID=0 PID=1931050 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:48:44 UTC] USER=www-data EUID=0 PID=1931065 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:44 UTC] USER=www-data EUID=0 PID=1931148 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:45 UTC] USER=www-data EUID=0 PID=1931223 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 10:48:45 UTC] USER=www-data EUID=0 PID=1931328 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 10:48:45 UTC] USER=www-data EUID=0 PID=1931356 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:48:45 UTC] USER=www-data EUID=0 PID=1931393 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:48:45 UTC] USER=www-data EUID=0 PID=1931412 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 10:48:45 UTC] USER=www-data EUID=0 PID=1931433 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 10:48:45 UTC] USER=www-data EUID=0 PID=1931460 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:46 UTC] USER=www-data EUID=0 PID=1931528 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:46 UTC] USER=www-data EUID=0 PID=1931608 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:48:46 UTC] USER=www-data EUID=0 PID=1931639 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:48:46 UTC] USER=www-data EUID=0 PID=1931666 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:46 UTC] USER=www-data EUID=0 PID=1931694 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:46 UTC] USER=www-data EUID=0 PID=1931717 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 10:48:47 UTC] USER=www-data EUID=0 PID=1931796 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:48:47 UTC] USER=www-data EUID=0 PID=1931849 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:48:47 UTC] USER=www-data EUID=0 PID=1931885 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 10:48:47 UTC] USER=www-data EUID=0 PID=1931903 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 10:48:47 UTC] USER=www-data EUID=0 PID=1931928 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:47 UTC] USER=www-data EUID=0 PID=1931963 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:48 UTC] USER=www-data EUID=0 PID=1932034 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:48:48 UTC] USER=www-data EUID=0 PID=1932104 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:48:48 UTC] USER=www-data EUID=0 PID=1932142 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:48 UTC] USER=www-data EUID=0 PID=1932220 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:48 UTC] USER=www-data EUID=0 PID=1932263 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 10:48:48 UTC] USER=www-data EUID=0 PID=1932333 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:48:48 UTC] USER=www-data EUID=0 PID=1932355 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:48:49 UTC] USER=www-data EUID=0 PID=1932378 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 10:48:49 UTC] USER=www-data EUID=0 PID=1932430 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 10:48:49 UTC] USER=www-data EUID=0 PID=1932460 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:50 UTC] USER=www-data EUID=0 PID=1932591 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:50 UTC] USER=www-data EUID=0 PID=1932636 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:48:50 UTC] USER=www-data EUID=0 PID=1932700 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:48:50 UTC] USER=www-data EUID=0 PID=1932733 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:50 UTC] USER=www-data EUID=0 PID=1932756 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:48:50 UTC] USER=www-data EUID=0 PID=1932773 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-02 10:48:50 UTC] USER=www-data EUID=0 PID=1932800 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-02 10:48:51 UTC] USER=www-data EUID=0 PID=1932823 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:48:51 UTC] USER=www-data EUID=0 PID=1932928 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:48:51 UTC] USER=www-data EUID=0 PID=1932993 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 10:48:51 UTC] USER=www-data EUID=0 PID=1933035 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-02 10:48:51 UTC] USER=www-data EUID=0 PID=1933081 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:48:51 UTC] USER=www-data EUID=0 PID=1933118 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:48:52 UTC] USER=www-data EUID=0 PID=1933297 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 10:48:52 UTC] USER=www-data EUID=0 PID=1933380 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 10:48:52 UTC] USER=www-data EUID=0 PID=1933445 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 10:48:52 UTC] USER=www-data EUID=0 PID=1933475 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:48:52 UTC] USER=www-data EUID=0 PID=1933508 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:48:52 UTC] USER=www-data EUID=0 PID=1933557 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:48:53 UTC] USER=www-data EUID=0 PID=1933621 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 📦 02 setup pg instance...
[0;34m[DEADLOCK-PREVENTION][0m Deadlock prevention library loaded
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-coordinator-postgresql environment: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com (10.100.1.213)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.213
[0;34m[INFO][0m Environment: identity-sau-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m Data dir:   /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[0;34m[INFO][0m Port:       5432
[0;34m[INFO][0m Hostname:   db-identity-sau-main-dev-postgresql-coordinator
[2026-01-02 10:48:57 UTC] USER=www-data EUID=0 PID=1935136 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:48:57 UTC] USER=www-data EUID=0 PID=1935157 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:48:57 UTC] USER=www-data EUID=0 PID=1935178 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[1;33m[WARN][0m Server certificate not found at /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[0;34m[INFO][0m Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 10:48:59 UTC] USER=www-data EUID=0 PID=1935246 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator and /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-02 10:48:59 UTC] USER=www-data EUID=0 PID=1935265 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1935206
[2026-01-02 10:48:59 UTC] USER=www-data EUID=0 PID=1935274 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1935206/ra_root.crt
[2026-01-02 10:49:00 UTC] USER=www-data EUID=0 PID=1935292 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1935206/ra_root.crt
[2026-01-02 10:49:00 UTC] USER=www-data EUID=0 PID=1935301 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1935206/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-02 10:49:04 UTC] USER=www-data EUID=0 PID=1935360 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1935206/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 10:49:04 UTC] USER=www-data EUID=0 PID=1935369 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1935206/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 10:49:04 UTC] USER=www-data EUID=0 PID=1935378 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-02 10:49:04 UTC] USER=www-data EUID=0 PID=1935387 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1935206/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:49:04 UTC] USER=www-data EUID=0 PID=1935396 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:49:04 UTC] USER=www-data EUID=0 PID=1935405 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:49:04 UTC] USER=www-data EUID=0 PID=1935414 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 10:49:04 UTC] USER=www-data EUID=0 PID=1935425 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 10:49:04 UTC] USER=www-data EUID=0 PID=1935435 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-02 10:49:04 UTC] USER=www-data EUID=0 PID=1935444 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 10:49:04 UTC] USER=www-data EUID=0 PID=1935453 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 10:49:04 UTC] USER=www-data EUID=0 PID=1935462 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:49:04 UTC] USER=www-data EUID=0 PID=1935471 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        coordinator
Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-02 10:49:05 UTC] USER=www-data EUID=0 PID=1935500 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-02 10:49:05 UTC] USER=www-data EUID=0 PID=1935509 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[0;32m[OK][0m   mTLS certificates OK (server cert + client certs verified) and keys secured
[0;34m[INFO][0m Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-02 10:49:05 UTC] USER=www-data EUID=0 PID=1935539 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 10:49:05 UTC] USER=www-data EUID=0 PID=1935563 ACTION=passthru ARGS=systemctl stop postgresql
[1;33m[WARN][0m Cleaning stale socket directory /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-02 10:49:06 UTC] USER=www-data EUID=0 PID=1935594 ACTION=fsop ARGS=rm -rf /var/run/postgresql-identity-sau-main-dev-coordinator
[0;32m[OK][0m   No conflicting Postgres left on port 5432
[0;32m[OK][0m   Using postgres password from vault provider
[2026-01-02 10:49:10 UTC] USER=www-data EUID=0 PID=1935653 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.fOIFiO
[2026-01-02 10:49:10 UTC] USER=www-data EUID=0 PID=1935674 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.fOIFiO
[2026-01-02 10:49:10 UTC] USER=www-data EUID=0 PID=1935696 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 10:49:10 UTC] USER=www-data EUID=0 PID=1935718 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 10:49:10 UTC] USER=www-data EUID=0 PID=1935740 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/identity-sau-main-dev
[0;34m[INFO][0m Initializing cluster in /var/lib/postgresql/17/identity-sau-main-dev/coordinator (SCRAM; pwfile)
[1;33m[WARN][0m Removing existing data directory: /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 10:49:10 UTC] USER=www-data EUID=0 PID=1935761 ACTION=fsop ARGS=rm -rf /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 10:49:11 UTC] USER=www-data EUID=0 PID=1935783 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 10:49:11 UTC] USER=www-data EUID=0 PID=1935804 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 10:49:11 UTC] USER=www-data EUID=0 PID=1935826 ACTION=fsop ARGS=chmod 700 /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-02 10:49:11 UTC] USER=www-data EUID=0 PID=1935848 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-02 10:49:11 UTC] USER=www-data EUID=0 PID=1935869 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-02 10:49:12 UTC] USER=www-data EUID=0 PID=1935890 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-02 10:49:12 UTC] USER=www-data EUID=0 PID=1935899 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /var/lib/postgresql/17/identity-sau-main-dev/coordinator --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.fOIFiO
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/17/identity-sau-main-dev/coordinator ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /var/lib/postgresql/17/identity-sau-main-dev/coordinator -l logfile start

[0;32m[OK][0m   initdb complete
[2026-01-02 10:49:14 UTC] USER=www-data EUID=0 PID=1935938 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.fOIFiO
[0;34m[INFO][0m Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[0;32m[OK][0m   postgresql.conf updated successfully
[0;34m[INFO][0m Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-02 10:49:14 UTC] USER=www-data EUID=0 PID=1935985 ACTION=fsop ARGS=cp /tmp/tmp.ZOsjxGtCFY /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
[2026-01-02 10:49:14 UTC] USER=www-data EUID=0 PID=1936006 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
[2026-01-02 10:49:14 UTC] USER=www-data EUID=0 PID=1936027 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
[0;32m[OK][0m   pg_hba.conf updated
[0;34m[INFO][0m Creating systemd unit: /etc/systemd/system/postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 10:49:14 UTC] USER=www-data EUID=0 PID=1936052 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.lbX4X2 /etc/systemd/system/postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 10:49:14 UTC] USER=www-data EUID=0 PID=1936073 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@identity-sau-main-dev-coordinator.service
[0;32m[OK][0m   systemd unit written
[2026-01-02 10:49:15 UTC] USER=www-data EUID=0 PID=1936094 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 10:49:15 UTC] USER=www-data EUID=0 PID=1936115 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 10:49:15 UTC] USER=www-data EUID=0 PID=1936136 ACTION=passthru ARGS=systemctl daemon-reload
[0;34m[INFO][0m Starting PostgreSQL instance...
[2026-01-02 10:49:17 UTC] USER=www-data EUID=0 PID=1936251 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[0;34m[INFO][0m Waiting for ACTIVE (systemd)…
[2026-01-02 10:49:17 UTC] USER=www-data EUID=0 PID=1936292 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[0;32m[OK][0m   Service ACTIVE
[0;34m[INFO][0m Waiting for port 5432 bind…
[0;32m[OK][0m   Port bound
[0;34m[INFO][0m Waiting pg_isready (socket)…
[0;32m[OK][0m   Readiness via socket OK
[0;34m[INFO][0m Waiting pg_isready (TCP db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432)…
[0;32m[OK][0m   Startup sequence complete
[0;34m[INFO][0m Validating core security GUCs (via local socket)…
[0;32m[OK][0m   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[0;34m[INFO][0m Provisioning application database and Debezium role (if not exists)...
[0;34m[INFO][0m Checking if database fastorder_identity_sau_main_dev_db exists...
[0;34m[INFO][0m DB check result: exit_code=0, output='[2026-01-02 10:49:19 UTC] USER=www-data EUID=0 PID=1936451 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_identity_sau_main_dev_db''
[0;34m[INFO][0m Creating database fastorder_identity_sau_main_dev_db...
[2026-01-02 10:49:19 UTC] USER=www-data EUID=0 PID=1936474 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_identity_sau_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[0;32m[OK][0m   Database fastorder_identity_sau_main_dev_db created
[0;34m[INFO][0m Checking if role debezium_user exists...
[0;34m[INFO][0m Role check result: exit_code=0, output='[2026-01-02 10:49:19 UTC] USER=www-data EUID=0 PID=1936498 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[0;34m[INFO][0m Creating role debezium_user...
[2026-01-02 10:49:20 UTC] USER=www-data EUID=0 PID=1936526 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'Nr016y+zV0UtM2pQO71lmv2Y';
CREATE ROLE
[0;32m[OK][0m   Role debezium_user created
[2026-01-02 10:49:20 UTC] USER=www-data EUID=0 PID=1936549 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_identity_sau_main_dev_db" TO debezium_user;
GRANT
[0;32m[OK][0m   Application DB (fastorder_identity_sau_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[0;34m[INFO][0m Applying connection and memory optimizations...
[0;34m[INFO][0m Current settings: max_connections=100, work_mem=4MB
[0;34m[INFO][0m Target settings (coordinator): max_connections=150, work_mem=8MB
[2026-01-02 10:49:21 UTC] USER=www-data EUID=0 PID=1936631 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 150;
ALTER SYSTEM
[2026-01-02 10:49:21 UTC] USER=www-data EUID=0 PID=1936658 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-02 10:49:21 UTC] USER=www-data EUID=0 PID=1936681 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[0;32m[OK][0m   Settings applied to postgresql.auto.conf
[2026-01-02 10:49:22 UTC] USER=www-data EUID=0 PID=1936696 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/standby.signal
[0;34m[INFO][0m Service recently started (5s ago) - restarting to apply max_connections...
[0;34m[INFO][0m Stopping service...
[2026-01-02 10:49:22 UTC] USER=www-data EUID=0 PID=1936722 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[0;34m[INFO][0m Waiting for port 5432 to be released...
[0;32m[OK][0m   Port 5432 released
[0;34m[INFO][0m Starting service...
[2026-01-02 10:49:25 UTC] USER=www-data EUID=0 PID=1937091 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 10:49:31 UTC] USER=www-data EUID=0 PID=1937147 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[0;32m[OK][0m   ✅ Optimization complete: max_connections=150, work_mem=8MB
[0;34m[INFO][0m Setting postgres password via centralized script... for coordinator
[0;34m[INFO][0m Temporarily disabling synchronous_commit on coordinator for password setting...
[0;32m[OK][0m   Disabled synchronous_commit (was: on)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    identity[0m
[0;34m  Zone:       sau[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: coordinator[0m

[0;34mAWS Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-identity-sau-main-dev-coordinator[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mInitial setup: Using password from initdb[0m
[0;32m✓ PostgreSQL password already set during initdb[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/coordinator
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/identity/sau/main/dev/postgresql/coordinator[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials coordinator[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
[0;34m[INFO][0m Restoring synchronous_commit on coordinator...
[0;32m[OK][0m   Restored synchronous_commit to: on
[0;32m[OK][0m   Password set and persisted
[0;34m[INFO][0m Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: identity-sau-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.213
[0;34m[INFO][0m Primary hostname: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

[0;34m[INFO][0m Adding /etc/hosts entries for coordinator...
[0;34m[INFO][0m   1. db-identity-sau-main-dev-postgresql.fastorder.com → 10.100.1.213 (primary/short)
[0;34m[INFO][0m   2. db-identity-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.213 (compatibility)

[0;34m[INFO][0m   ✅ db-identity-sau-main-dev-postgresql.fastorder.com already exists with correct IP
[0;34m[INFO][0m   ✅ db-identity-sau-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;32m✅[0m   ✅ Network & DNS configuration complete
[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
  10.100.1.213    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  10.100.1.213    db-identity-sau-main-dev-postgresql.fastorder.com


[0;32m[OK][0m   PostgreSQL 'identity-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key \
        host=db-identity-sau-main-dev-postgresql-coordinator port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[0;34m[INFO][0m Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.213
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-coordinator
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: ce097707-5ce5-40c8-a941-01512555cab8
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[0;32m[OK][0m   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 📦 03 role...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 10:49:47 UTC] USER=www-data EUID=0 PID=1937636 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    debezium_user
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   debezium_user
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 10:50:13 UTC] USER=www-data EUID=0 PID=1937921 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-01-02 10:50:13 UTC] USER=www-data EUID=0 PID=1937930 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-debezium_user/ra_root.key
[2026-01-02 10:50:13 UTC] USER=www-data EUID=0 PID=1937939 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-01-02 10:50:13 UTC] USER=www-data EUID=0 PID=1937948 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:50:14 UTC] USER=www-data EUID=0 PID=1937962 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:50:14 UTC] USER=www-data EUID=0 PID=1937971 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:50:14 UTC] USER=www-data EUID=0 PID=1937980 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 10:50:14 UTC] USER=www-data EUID=0 PID=1937989 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 10:50:14 UTC] USER=www-data EUID=0 PID=1937998 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:50:14 UTC] USER=www-data EUID=0 PID=1938007 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:50:14 UTC] USER=www-data EUID=0 PID=1938016 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 10:50:14 UTC] USER=www-data EUID=0 PID=1938025 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 10:50:14 UTC] USER=www-data EUID=0 PID=1938034 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 10:50:14 UTC] USER=www-data EUID=0 PID=1938043 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:50:14 UTC] USER=www-data EUID=0 PID=1938052 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:50:14 UTC] USER=www-data EUID=0 PID=1938061 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 10:50:14 UTC] USER=www-data EUID=0 PID=1938070 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 10:50:14 UTC] USER=www-data EUID=0 PID=1938079 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 10:50:14 UTC] USER=www-data EUID=0 PID=1938088 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:50:14 UTC] USER=www-data EUID=0 PID=1938098 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:14 UTC] USER=www-data EUID=0 PID=1938124 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:15 UTC] USER=www-data EUID=0 PID=1938133 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:50:15 UTC] USER=www-data EUID=0 PID=1938142 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:50:15 UTC] USER=www-data EUID=0 PID=1938151 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:15 UTC] USER=www-data EUID=0 PID=1938160 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:15 UTC] USER=www-data EUID=0 PID=1938169 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 10:50:15 UTC] USER=www-data EUID=0 PID=1938178 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 10:50:15 UTC] USER=www-data EUID=0 PID=1938187 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:50:15 UTC] USER=www-data EUID=0 PID=1938196 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:50:15 UTC] USER=www-data EUID=0 PID=1938205 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 10:50:15 UTC] USER=www-data EUID=0 PID=1938214 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 10:50:15 UTC] USER=www-data EUID=0 PID=1938224 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:15 UTC] USER=www-data EUID=0 PID=1938234 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:15 UTC] USER=www-data EUID=0 PID=1938243 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:50:15 UTC] USER=www-data EUID=0 PID=1938261 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:15 UTC] USER=www-data EUID=0 PID=1938270 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:15 UTC] USER=www-data EUID=0 PID=1938279 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 10:50:15 UTC] USER=www-data EUID=0 PID=1938288 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938297 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938306 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938315 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938324 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938336 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938346 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938355 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938364 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938373 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938391 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938400 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938410 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938420 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938429 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938438 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938448 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938458 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938467 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938476 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:50:16 UTC] USER=www-data EUID=0 PID=1938485 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:17 UTC] USER=www-data EUID=0 PID=1938494 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:17 UTC] USER=www-data EUID=0 PID=1938503 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-02 10:50:17 UTC] USER=www-data EUID=0 PID=1938512 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-02 10:50:17 UTC] USER=www-data EUID=0 PID=1938521 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:50:17 UTC] USER=www-data EUID=0 PID=1938530 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:50:17 UTC] USER=www-data EUID=0 PID=1938539 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 10:50:17 UTC] USER=www-data EUID=0 PID=1938548 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-02 10:50:17 UTC] USER=www-data EUID=0 PID=1938558 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:50:17 UTC] USER=www-data EUID=0 PID=1938570 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:50:17 UTC] USER=www-data EUID=0 PID=1938579 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:50:17 UTC] USER=www-data EUID=0 PID=1938588 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 10:50:17 UTC] USER=www-data EUID=0 PID=1938597 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 10:50:17 UTC] USER=www-data EUID=0 PID=1938606 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 10:50:17 UTC] USER=www-data EUID=0 PID=1938615 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:50:17 UTC] USER=www-data EUID=0 PID=1938624 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:50:17 UTC] USER=www-data EUID=0 PID=1938633 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:50:17 UTC] USER=www-data EUID=0 PID=1938642 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: debezium_user
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U debezium_user -d postgres

✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    identity[0m
[0;34m  Zone:       sau[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: coordinator[0m

[0;34mAWS Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-identity-sau-main-dev-coordinator[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mGenerating new secure password...[0m
[0;34mUser fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: fastorder_admin_gd[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials coordinator[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: coordinator/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-identity-sau-main-dev-postgresql.fastorder.com
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    fastorder_admin_gd
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   fastorder_admin_gd
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 10:50:35 UTC] USER=www-data EUID=0 PID=1939349 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-fastorder_admin_gd
[2026-01-02 10:50:35 UTC] USER=www-data EUID=0 PID=1939358 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-02 10:50:35 UTC] USER=www-data EUID=0 PID=1939367 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
[2026-01-02 10:50:35 UTC] USER=www-data EUID=0 PID=1939376 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-02 10:50:35 UTC] USER=www-data EUID=0 PID=1939385 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:50:35 UTC] USER=www-data EUID=0 PID=1939399 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:50:35 UTC] USER=www-data EUID=0 PID=1939408 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:50:35 UTC] USER=www-data EUID=0 PID=1939417 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 10:50:35 UTC] USER=www-data EUID=0 PID=1939426 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 10:50:35 UTC] USER=www-data EUID=0 PID=1939435 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:50:35 UTC] USER=www-data EUID=0 PID=1939444 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:50:35 UTC] USER=www-data EUID=0 PID=1939453 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 10:50:35 UTC] USER=www-data EUID=0 PID=1939462 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 10:50:35 UTC] USER=www-data EUID=0 PID=1939471 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 10:50:35 UTC] USER=www-data EUID=0 PID=1939480 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939489 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939498 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939507 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939516 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939525 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939534 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939543 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939552 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939578 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939587 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939597 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939607 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939616 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939625 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939634 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939643 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939652 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939661 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939670 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939680 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939690 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939700 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939710 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939719 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:36 UTC] USER=www-data EUID=0 PID=1939728 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939737 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939746 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939755 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939764 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939773 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939782 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939792 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939802 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939811 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939820 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939829 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939838 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939847 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939856 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939865 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939874 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939883 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939892 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939902 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939912 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939921 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939930 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939939 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939957 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939966 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939975 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-02 10:50:37 UTC] USER=www-data EUID=0 PID=1939985 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 10:50:38 UTC] USER=www-data EUID=0 PID=1939994 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 10:50:38 UTC] USER=www-data EUID=0 PID=1940003 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 10:50:38 UTC] USER=www-data EUID=0 PID=1940013 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:50:38 UTC] USER=www-data EUID=0 PID=1940023 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:50:38 UTC] USER=www-data EUID=0 PID=1940032 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:50:38 UTC] USER=www-data EUID=0 PID=1940041 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 10:50:38 UTC] USER=www-data EUID=0 PID=1940050 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 10:50:38 UTC] USER=www-data EUID=0 PID=1940063 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 10:50:38 UTC] USER=www-data EUID=0 PID=1940072 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:50:38 UTC] USER=www-data EUID=0 PID=1940081 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:50:38 UTC] USER=www-data EUID=0 PID=1940090 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:50:38 UTC] USER=www-data EUID=0 PID=1940099 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: fastorder_admin_gd
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-identity-sau-main-dev-coordinator:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_identity_sau_main_dev_db already exists, skipping creation
[2026-01-02 10:50:38 UTC] USER=www-data EUID=0 PID=1940157 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-02 10:50:39 UTC] USER=www-data EUID=0 PID=1940191 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

[0;34m=== Pre-flight Checks ===[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;32m✓[0m AWS Secrets Manager accessible

[0;34m=== Retrieving Credentials from AWS ===[0m
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
/opt/fastorder/bash/infra_core/cache.sh: line 145: /var/cache/secrets/fastorder_db_identity_sau_main_dev_postgresql_coordinator_fastorder_admin_gd.cache.tmp.1940200: Permission denied
✅ Retrieved from secrets manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials loaded for coordinator/fastorder_admin_gd: fastorder_admin_gd@db-identity-sau-main-dev-postgresql.fastorder.com:5432/fastorder_identity_sau_main_dev_db
[0;32m✓[0m Credentials retrieved: fastorder_admin_gd@db-identity-sau-main-dev-postgresql.fastorder.com:5432/fastorder_identity_sau_main_dev_db
[0;34m╔════════════════════════════════════════════╗[0m
[0;34m║  PostgreSQL Test Suite (AWS Secrets MGR)  ║[0m
[0;34m╚════════════════════════════════════════════╝[0m

[0;34m=== PostgreSQL Authentication Test ===[0m
[0;31m✗[0m PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-identity-sau-main-dev-postgresql.fastorder.com" (10.100.1.213), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 10:50:47 UTC] USER=www-data EUID=0 PID=1940377 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : coordinator
  PG HOST     : db-identity-sau-main-dev-postgresql.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_identity_sau_main_dev_db
  SCHEMA      : auth
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
  DNS → 10.100.1.213
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    identity[0m
[0;34m  Zone:       sau[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: coordinator[0m

[0;34mAWS Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-identity-sau-main-dev-coordinator[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mGenerating new secure password...[0m
[0;34mUser debezium_user does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: debezium_user[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials coordinator[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: coordinator/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-01-02 10:51:03 UTC] USER=www-data EUID=0 PID=1940710 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
[2026-01-02 10:51:03 UTC] USER=www-data EUID=0 PID=1940736 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d fastorder_identity_sau_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=coordinator IDENTIFIER_PARENT=coordinator
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)

[0;35m╔════════════════════════════════════════════════════════════════════════════╗[0m
[0;35m║                    IAM Database Schema Initialization                       ║[0m
[0;35m╚════════════════════════════════════════════════════════════════════════════╝[0m

[0;34m[INFO][0m 🟢 Starting IAM schema provisioning...
[0;34m[INFO][0m Environment: identity-sau-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m VM IP: 142.93.238.16

[0;34m[INFO][0m 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;34m  Schema: core[0m
[0;34m  Core Identity Directory (tenants, realms, identities, devices, MFA)[0m
[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m

[0;34m[INFO][0m 🔸 Table [1/20]: core/01-tenant
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
🔧 Installing extensions...
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
   Creating reference table: core.tenant
 create_reference_table 
------------------------
 
(1 row)

✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[0;32m[OK][0m Table core/01-tenant initialized

[0;34m[INFO][0m 🔸 Table [2/20]: core/02-realm
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.realm$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_realm_updated" for relation "core.realm" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[0;32m[OK][0m Table core/02-realm initialized

[0;34m[INFO][0m 🔸 Table [3/20]: core/03-identity
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_identity_updated" for relation "core.identity" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[0;32m[OK][0m Table core/03-identity initialized

[0;34m[INFO][0m 🔸 Table [4/20]: core/04-device
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.device$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[0;32m[OK][0m Table core/04-device initialized

[0;34m[INFO][0m 🔸 Table [5/20]: core/05-identity_account
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity_account$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_identity_account_updated" for relation "core.identity_account" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[0;32m[OK][0m Table core/05-identity_account initialized

[0;34m[INFO][0m 🔸 Table [6/20]: core/06-identity_mfa
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity_mfa$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[0;32m[OK][0m Table core/06-identity_mfa initialized

[0;34m[INFO][0m 🔸 Table [7/20]: core/07-external_idp_link
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.external_idp_link$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[0;32m[OK][0m Table core/07-external_idp_link initialized


[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;34m  Schema: policy[0m
[0;34m  RBAC/ABAC Authorization (clients, roles, permissions, policies)[0m
[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m

[0;34m[INFO][0m 🔸 Table [8/20]: policy/01-client
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.client$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
NOTICE:  trigger "tr_client_updated" for relation "policy.client" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[0;32m[OK][0m Table policy/01-client initialized

[0;34m[INFO][0m 🔸 Table [9/20]: policy/02-resource
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.resource$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[0;32m[OK][0m Table policy/02-resource initialized

[0;34m[INFO][0m 🔸 Table [10/20]: policy/03-scope
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.scope$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_scope_updated" for relation "policy.scope" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[0;32m[OK][0m Table policy/03-scope initialized

[0;34m[INFO][0m 🔸 Table [11/20]: policy/04-permission
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.permission$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_permission_updated" for relation "policy.permission" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[0;32m[OK][0m Table policy/04-permission initialized

[0;34m[INFO][0m 🔸 Table [12/20]: policy/05-role
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.role$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_role_updated" for relation "policy.role" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[0;32m[OK][0m Table policy/05-role initialized

[0;34m[INFO][0m 🔸 Table [13/20]: policy/06-role_permission
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.role_permission$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[0;32m[OK][0m Table policy/06-role_permission initialized

[0;34m[INFO][0m 🔸 Table [14/20]: policy/07-identity_role
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.identity_role$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[0;32m[OK][0m Table policy/07-identity_role initialized

[0;34m[INFO][0m 🔸 Table [15/20]: policy/08-policy_rule
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.policy_rule$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_policy_rule_updated" for relation "policy.policy_rule" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[0;32m[OK][0m Table policy/08-policy_rule initialized

[0;34m[INFO][0m 🔸 Table [16/20]: policy/09-api_key
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.api_key$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[0;32m[OK][0m Table policy/09-api_key initialized


[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;34m  Schema: audit[0m
[0;34m  Audit & Risk Logging (auth events, admin actions, risk decisions)[0m
[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m

[0;34m[INFO][0m 🔸 Table [17/20]: audit/01-auth_event
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[0;32m[OK][0m Table audit/01-auth_event initialized

[0;34m[INFO][0m 🔸 Table [18/20]: audit/02-admin_action
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[0;32m[OK][0m Table audit/02-admin_action initialized

[0;34m[INFO][0m 🔸 Table [19/20]: audit/03-risk_decision
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[0;32m[OK][0m Table audit/03-risk_decision initialized

[0;34m[INFO][0m 🔸 Table [20/20]: audit/04-consent_event
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[0;32m[OK][0m Table audit/04-consent_event initialized


[0;35m════════════════════════════════════════════════════════════════════════════[0m
[0;32m[OK][0m ✅ IAM Schema Initialization Complete!
[0;32m[OK][0m All 20 tables initialized successfully

[0;34mSchemas created:[0m
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

[0;34mDesign highlights:[0m
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

[0;35m════════════════════════════════════════════════════════════════════════════[0m

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=coordinator IDENTIFIER_PARENT=coordinator
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: identity/run.sh started for SERVICE=identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity
🔍 DEBUG_CHECKPOINT_A3: SERVICE_ROOT exists, discovering table folders
🔍 DEBUG_CHECKPOINT_A4: Found subfolder: auth
🔍 DEBUG_CHECKPOINT_A4b: Checking for nested schema layout in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth
🔍 DEBUG_CHECKPOINT_A4c: Found nested steps dir: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps (display: auth/login)
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: auth/login|/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
[INFO] 📚 Detected grouped table folders under identity/: auth/login

🔍 DEBUG_CHECKPOINT_A7: Current IDENTIFIER=coordinator
🔍 DEBUG_CHECKPOINT_A8_PROCEED: Processing tables on coordinator/main node
🔍 DEBUG_CHECKPOINT_A9: Processing table: auth/login at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
[INFO] 🔸 Table group: auth/login
🔍 DEBUG_CHECKPOINT_A10: About to run numbered steps for table: auth/login
🔍 DEBUG_CHECKPOINT_B1: run_all_numbered_steps_in_dir called for dir=/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps table=auth/login
🔍 DEBUG_CHECKPOINT_B2: Found 1 numbered steps: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B3: About to run step: 01-init-schema.sh
Ab substep 0 compelete start
[DEBUG] Tracking substep start: steps/01-install/steps/identity/auth/login/01-init-schema (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
Ab substep 0 compelete start
[INFO] 📦 01 init schema...
Ab substep 1 compelete start
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing auth.login_account table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating auth schema...
NOTICE:  schema "auth" already exists, skipping
CREATE SCHEMA
✅ Schema created
🔧 Creating account_status ENUM...
DO
✅ ENUM created
🔧 Creating auth.login_account table...
CREATE TABLE
✅ Table created (Citus-compatible with region_hint in all constraints)
🔧 Creating indexes...
CREATE INDEX
CREATE INDEX
✅ Indexes created
🔧 Creating Citus REFERENCE table for CDC compatibility...
 create_reference_table 
------------------------
 
(1 row)

✅ Table created as REFERENCE table (replicated to all nodes)
   CDC via Debezium will work correctly on coordinator
🎉 Schema initialization complete for fastorder_identity_sau_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

📊 Registering environment in monitoring database (obs schema)...
   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
   Resource IP: 142.93.238.16
⚠️  Could not connect to monitoring database, skipping registration
   You can manually register later using:
   /opt/fastorder/bash/scripts/env_app_setup/setup/04-postgresql/steps/register-authN-af-aaaa1-dev.sh

==========================================
✅ Schema initialization complete!
==========================================
Ab substep 1 compelete end
Ab substep 2 compelete start
Ab substep 2 compelete end

🔍 DEBUG_CHECKPOINT_B4: Completed step: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B5: All numbered steps completed for /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A11: Completed numbered steps for table: auth/login
compeleted here

🔍 DEBUG_CHECKPOINT_A12: All tables processed
End of 04-postgresql/steps/01-install/steps/identity/run.sh

[0;32m✓[0m ✅ Coordinator setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up 1 worker(s) (Citus data nodes)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up worker: worker-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-02 10:52:49 UTC] USER=unknown EUID=33 PID=1944607 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 10:52:49 UTC] USER=unknown EUID=33 PID=1944615 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 10:52:49 UTC] USER=unknown EUID=33 PID=1944622 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-02 10:52:49 UTC] USER=unknown EUID=33 PID=1944636 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-02 10:52:49 UTC] USER=unknown EUID=33 PID=1944629 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 10:52:49 UTC] USER=unknown EUID=33 PID=1944643 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede JOB_UUID=333bba61-dfcf-4fcb-986b-f0541277e251

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: identity-sau-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.214
[0;34m[INFO][0m Primary hostname: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

[0;34m[INFO][0m Adding /etc/hosts entry for worker-01...
[0;34m[INFO][0m   db-identity-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.214

[0;34m[INFO][0m   ✅ db-identity-sau-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;32m✅[0m   ✅ Network & DNS configuration complete
[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
  10.100.1.214    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 10:52:53 UTC] USER=www-data EUID=0 PID=1944787 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:52:53 UTC] USER=www-data EUID=0 PID=1944796 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-02 10:52:53 UTC] USER=www-data EUID=0 PID=1944806 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1944747
[2026-01-02 10:52:53 UTC] USER=www-data EUID=0 PID=1944815 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1944747/ra_root.crt
[2026-01-02 10:52:54 UTC] USER=www-data EUID=0 PID=1944824 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1944747/ra_root.key
[2026-01-02 10:52:54 UTC] USER=www-data EUID=0 PID=1944833 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1944747/ra_root.crt
[2026-01-02 10:52:54 UTC] USER=www-data EUID=0 PID=1944842 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1944747/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[2026-01-02 10:52:56 UTC] USER=www-data EUID=0 PID=1944895 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1944747/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 10:52:56 UTC] USER=www-data EUID=0 PID=1944904 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1944747/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 10:52:56 UTC] USER=www-data EUID=0 PID=1944915 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 10:52:56 UTC] USER=www-data EUID=0 PID=1944924 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1944747/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:52:56 UTC] USER=www-data EUID=0 PID=1944933 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:52:56 UTC] USER=www-data EUID=0 PID=1944942 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:52:56 UTC] USER=www-data EUID=0 PID=1944951 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 10:52:56 UTC] USER=www-data EUID=0 PID=1944962 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 10:52:56 UTC] USER=www-data EUID=0 PID=1944971 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 10:52:56 UTC] USER=www-data EUID=0 PID=1944980 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 10:52:56 UTC] USER=www-data EUID=0 PID=1944989 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 10:52:56 UTC] USER=www-data EUID=0 PID=1944998 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:52:56 UTC] USER=www-data EUID=0 PID=1945007 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01
Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 10:52:57 UTC] USER=www-data EUID=0 PID=1945061 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-01-02 10:52:57 UTC] USER=www-data EUID=0 PID=1945070 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 10:52:57 UTC] USER=www-data EUID=0 PID=1945079 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-01-02 10:52:57 UTC] USER=www-data EUID=0 PID=1945088 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 10:52:57 UTC] USER=www-data EUID=0 PID=1945097 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:52:57 UTC] USER=www-data EUID=0 PID=1945111 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:52:57 UTC] USER=www-data EUID=0 PID=1945120 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:52:57 UTC] USER=www-data EUID=0 PID=1945131 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 10:52:57 UTC] USER=www-data EUID=0 PID=1945140 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 10:52:57 UTC] USER=www-data EUID=0 PID=1945149 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:52:57 UTC] USER=www-data EUID=0 PID=1945158 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:52:57 UTC] USER=www-data EUID=0 PID=1945178 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 10:52:57 UTC] USER=www-data EUID=0 PID=1945187 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 10:52:57 UTC] USER=www-data EUID=0 PID=1945196 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:52:57 UTC] USER=www-data EUID=0 PID=1945205 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:52:57 UTC] USER=www-data EUID=0 PID=1945214 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 10:52:57 UTC] USER=www-data EUID=0 PID=1945223 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945232 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945241 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945252 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945279 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945288 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945297 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945306 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945317 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945328 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945337 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945346 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945355 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945364 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945373 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945384 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945394 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945403 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945412 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945421 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945430 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945439 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945448 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 10:52:58 UTC] USER=www-data EUID=0 PID=1945457 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945466 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945475 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945484 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945494 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945504 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945513 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945522 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945531 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945540 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945549 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945558 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945567 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945576 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945585 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945594 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945604 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945614 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945623 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945633 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945643 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945652 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945661 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945670 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945679 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945688 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945697 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945706 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 10:52:59 UTC] USER=www-data EUID=0 PID=1945716 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:53:00 UTC] USER=www-data EUID=0 PID=1945726 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:53:00 UTC] USER=www-data EUID=0 PID=1945736 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:53:00 UTC] USER=www-data EUID=0 PID=1945746 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 10:53:00 UTC] USER=www-data EUID=0 PID=1945755 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 10:53:00 UTC] USER=www-data EUID=0 PID=1945764 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 10:53:00 UTC] USER=www-data EUID=0 PID=1945773 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:53:00 UTC] USER=www-data EUID=0 PID=1945782 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:53:00 UTC] USER=www-data EUID=0 PID=1945791 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:53:00 UTC] USER=www-data EUID=0 PID=1945800 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 10:53:01 UTC] USER=www-data EUID=0 PID=1945842 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-01-02 10:53:01 UTC] USER=www-data EUID=0 PID=1945852 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 10:53:01 UTC] USER=www-data EUID=0 PID=1945861 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-01-02 10:53:01 UTC] USER=www-data EUID=0 PID=1945870 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 10:53:01 UTC] USER=www-data EUID=0 PID=1945879 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:53:01 UTC] USER=www-data EUID=0 PID=1945908 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1945917 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1945926 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1945935 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1945944 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1945954 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1945965 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1945974 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1945983 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1945992 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1946001 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1946010 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1946019 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1946028 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1946037 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1946046 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1946055 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1946064 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1946090 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1946099 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1946110 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1946120 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1946130 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1946139 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 10:53:02 UTC] USER=www-data EUID=0 PID=1946148 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946157 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946166 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946175 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946184 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946194 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946204 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946214 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946224 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946233 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946244 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946255 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946264 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946273 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946282 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946291 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946300 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946310 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946320 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946329 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946338 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946347 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946356 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:53:03 UTC] USER=www-data EUID=0 PID=1946365 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946374 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946383 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946392 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946401 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946410 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946420 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946430 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946439 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946448 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946457 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946466 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946475 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946484 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946493 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946502 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946511 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946520 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946530 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946540 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946551 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946560 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 10:53:04 UTC] USER=www-data EUID=0 PID=1946569 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
   ✅ Symlinked client-key.pem
[2026-01-02 10:53:05 UTC] USER=www-data EUID=0 PID=1946587 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:53:05 UTC] USER=www-data EUID=0 PID=1946596 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:53:05 UTC] USER=www-data EUID=0 PID=1946605 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:53:05 UTC] USER=www-data EUID=0 PID=1946614 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 📦 02 setup pg instance...
[0;34m[DEADLOCK-PREVENTION][0m Deadlock prevention library loaded
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-postgresql environment: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com (10.100.1.214)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.214
[0;34m[INFO][0m Environment: identity-sau-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m Data dir:   /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[0;34m[INFO][0m Port:       5432
[0;34m[INFO][0m Hostname:   db-identity-sau-main-dev-postgresql-worker-01
[2026-01-02 10:53:07 UTC] USER=www-data EUID=0 PID=1946722 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:53:07 UTC] USER=www-data EUID=0 PID=1946743 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:53:07 UTC] USER=www-data EUID=0 PID=1946764 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:53:07 UTC] USER=www-data EUID=0 PID=1946785 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[1;33m[WARN][0m Server certificate not found at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[0;34m[INFO][0m Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 10:53:08 UTC] USER=www-data EUID=0 PID=1946827 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:53:08 UTC] USER=www-data EUID=0 PID=1946837 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-02 10:53:08 UTC] USER=www-data EUID=0 PID=1946850 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1946792
[2026-01-02 10:53:08 UTC] USER=www-data EUID=0 PID=1946859 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1946792/ra_root.crt
[2026-01-02 10:53:08 UTC] USER=www-data EUID=0 PID=1946868 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1946792/ra_root.key
[2026-01-02 10:53:08 UTC] USER=www-data EUID=0 PID=1946877 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1946792/ra_root.crt
[2026-01-02 10:53:08 UTC] USER=www-data EUID=0 PID=1946886 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1946792/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[2026-01-02 10:53:10 UTC] USER=www-data EUID=0 PID=1946928 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1946792/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 10:53:10 UTC] USER=www-data EUID=0 PID=1946937 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1946792/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 10:53:10 UTC] USER=www-data EUID=0 PID=1946946 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 10:53:10 UTC] USER=www-data EUID=0 PID=1946955 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1946792/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:53:10 UTC] USER=www-data EUID=0 PID=1946964 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:53:10 UTC] USER=www-data EUID=0 PID=1946973 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:53:10 UTC] USER=www-data EUID=0 PID=1946982 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 10:53:10 UTC] USER=www-data EUID=0 PID=1946993 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 10:53:10 UTC] USER=www-data EUID=0 PID=1947002 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 10:53:10 UTC] USER=www-data EUID=0 PID=1947020 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 10:53:10 UTC] USER=www-data EUID=0 PID=1947029 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:53:10 UTC] USER=www-data EUID=0 PID=1947038 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01
Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-02 10:53:10 UTC] USER=www-data EUID=0 PID=1947067 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-02 10:53:10 UTC] USER=www-data EUID=0 PID=1947076 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-02 10:53:10 UTC] USER=www-data EUID=0 PID=1947085 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[0;32m[OK][0m   mTLS certificates OK (server cert + client certs verified) and keys secured
[0;34m[INFO][0m Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-02 10:53:10 UTC] USER=www-data EUID=0 PID=1947108 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-worker-01.service
[2026-01-02 10:53:11 UTC] USER=www-data EUID=0 PID=1947132 ACTION=passthru ARGS=systemctl stop postgresql
[1;33m[WARN][0m Cleaning stale socket directory /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-02 10:53:11 UTC] USER=www-data EUID=0 PID=1947165 ACTION=fsop ARGS=rm -rf /var/run/postgresql-identity-sau-main-dev-worker-01
[0;32m[OK][0m   No conflicting Postgres left on port 5432
[0;32m[OK][0m   Using postgres password from vault provider
[2026-01-02 10:53:15 UTC] USER=www-data EUID=0 PID=1947233 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.VNH0le
[2026-01-02 10:53:15 UTC] USER=www-data EUID=0 PID=1947254 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.VNH0le
[2026-01-02 10:53:15 UTC] USER=www-data EUID=0 PID=1947276 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 10:53:15 UTC] USER=www-data EUID=0 PID=1947298 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 10:53:15 UTC] USER=www-data EUID=0 PID=1947320 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/identity-sau-main-dev
[0;34m[INFO][0m Initializing cluster in /var/lib/postgresql/17/identity-sau-main-dev/worker-01 (SCRAM; pwfile)
[1;33m[WARN][0m Removing existing data directory: /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 10:53:15 UTC] USER=www-data EUID=0 PID=1947341 ACTION=fsop ARGS=rm -rf /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 10:53:16 UTC] USER=www-data EUID=0 PID=1947363 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 10:53:16 UTC] USER=www-data EUID=0 PID=1947386 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 10:53:16 UTC] USER=www-data EUID=0 PID=1947407 ACTION=fsop ARGS=chmod 700 /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-02 10:53:16 UTC] USER=www-data EUID=0 PID=1947428 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-02 10:53:16 UTC] USER=www-data EUID=0 PID=1947449 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-02 10:53:16 UTC] USER=www-data EUID=0 PID=1947472 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-02 10:53:16 UTC] USER=www-data EUID=0 PID=1947481 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /var/lib/postgresql/17/identity-sau-main-dev/worker-01 --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.VNH0le
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/17/identity-sau-main-dev/worker-01 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /var/lib/postgresql/17/identity-sau-main-dev/worker-01 -l logfile start

[0;32m[OK][0m   initdb complete
[2026-01-02 10:53:18 UTC] USER=www-data EUID=0 PID=1947521 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.VNH0le
[0;34m[INFO][0m Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[0;32m[OK][0m   postgresql.conf updated successfully
[0;34m[INFO][0m Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-02 10:53:18 UTC] USER=www-data EUID=0 PID=1947573 ACTION=fsop ARGS=cp /tmp/tmp.GgoAcZ2FVT /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 10:53:18 UTC] USER=www-data EUID=0 PID=1947594 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 10:53:18 UTC] USER=www-data EUID=0 PID=1947617 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[0;32m[OK][0m   pg_hba.conf updated
[0;34m[INFO][0m Creating systemd unit: /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01.service
[2026-01-02 10:53:18 UTC] USER=www-data EUID=0 PID=1947642 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.9iJdV0 /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01.service
[2026-01-02 10:53:19 UTC] USER=www-data EUID=0 PID=1947663 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01.service
[0;32m[OK][0m   systemd unit written
[2026-01-02 10:53:19 UTC] USER=www-data EUID=0 PID=1947684 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 10:53:19 UTC] USER=www-data EUID=0 PID=1947705 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 10:53:19 UTC] USER=www-data EUID=0 PID=1947726 ACTION=passthru ARGS=systemctl daemon-reload
[0;34m[INFO][0m Starting PostgreSQL instance...
[2026-01-02 10:53:21 UTC] USER=www-data EUID=0 PID=1947847 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-worker-01.service
[0;34m[INFO][0m Waiting for ACTIVE (systemd)…
[2026-01-02 10:53:22 UTC] USER=www-data EUID=0 PID=1947887 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-worker-01.service
[0;32m[OK][0m   Service ACTIVE
[0;34m[INFO][0m Waiting for port 5432 bind…
[0;32m[OK][0m   Port bound
[0;34m[INFO][0m Waiting pg_isready (socket)…
[0;32m[OK][0m   Readiness via socket OK
[0;34m[INFO][0m Waiting pg_isready (TCP db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432)…
[0;32m[OK][0m   Startup sequence complete
[0;34m[INFO][0m Validating core security GUCs (via local socket)…
[0;32m[OK][0m   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[0;34m[INFO][0m Provisioning application database and Debezium role (if not exists)...
[0;34m[INFO][0m Checking if database fastorder_identity_sau_main_dev_db exists...
[0;34m[INFO][0m DB check result: exit_code=0, output='[2026-01-02 10:53:23 UTC] USER=www-data EUID=0 PID=1948052 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_identity_sau_main_dev_db''
[0;34m[INFO][0m Creating database fastorder_identity_sau_main_dev_db...
[2026-01-02 10:53:23 UTC] USER=www-data EUID=0 PID=1948076 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_identity_sau_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[0;32m[OK][0m   Database fastorder_identity_sau_main_dev_db created
[0;34m[INFO][0m Checking if role debezium_user exists...
[0;34m[INFO][0m Role check result: exit_code=0, output='[2026-01-02 10:53:23 UTC] USER=www-data EUID=0 PID=1948100 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[0;34m[INFO][0m Creating role debezium_user...
[2026-01-02 10:53:24 UTC] USER=www-data EUID=0 PID=1948127 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'pYRzLHv6s3tjDqNIppjOSy57';
CREATE ROLE
[0;32m[OK][0m   Role debezium_user created
[2026-01-02 10:53:24 UTC] USER=www-data EUID=0 PID=1948153 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_identity_sau_main_dev_db" TO debezium_user;
GRANT
[0;32m[OK][0m   Application DB (fastorder_identity_sau_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[0;34m[INFO][0m Applying connection and memory optimizations...
[0;34m[INFO][0m Current settings: max_connections=100, work_mem=4MB
[0;34m[INFO][0m Target settings (worker): max_connections=100, work_mem=8MB
[2026-01-02 10:53:24 UTC] USER=www-data EUID=0 PID=1948233 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 100;
ALTER SYSTEM
[2026-01-02 10:53:25 UTC] USER=www-data EUID=0 PID=1948258 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-02 10:53:25 UTC] USER=www-data EUID=0 PID=1948377 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[0;32m[OK][0m   Settings applied to postgresql.auto.conf
[2026-01-02 10:53:25 UTC] USER=www-data EUID=0 PID=1948621 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01/standby.signal
[0;34m[INFO][0m Service recently started (4s ago) - restarting to apply max_connections...
[0;34m[INFO][0m Stopping service...
[2026-01-02 10:53:26 UTC] USER=www-data EUID=0 PID=1948646 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-worker-01.service
[0;34m[INFO][0m Waiting for port 5432 to be released...
[0;32m[OK][0m   Port 5432 released
[0;34m[INFO][0m Starting service...
[2026-01-02 10:53:29 UTC] USER=www-data EUID=0 PID=1948692 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-worker-01.service
[2026-01-02 10:53:35 UTC] USER=www-data EUID=0 PID=1948761 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-worker-01.service
[0;32m[OK][0m   ✅ Optimization complete: max_connections=100, work_mem=8MB
[0;32m[OK][0m   Synchronous replication already configured (synchronous_commit: on)
[0;34m[INFO][0m Setting postgres password via centralized script... for worker-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    identity[0m
[0;34m  Zone:       sau[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: worker-01[0m

[0;34mAWS Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-identity-sau-main-dev-worker-01[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mInitial setup: Using password from initdb[0m
[0;32m✓ PostgreSQL password already set during initdb[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/worker-01
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/identity/sau/main/dev/postgresql/worker-01[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials worker-01[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
[0;32m[OK][0m   Password set and persisted
[0;34m[INFO][0m Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: identity-sau-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.214
[0;34m[INFO][0m Primary hostname: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

[0;34m[INFO][0m Adding /etc/hosts entry for worker-01...
[0;34m[INFO][0m   db-identity-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.214

[0;34m[INFO][0m   ✅ db-identity-sau-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;32m✅[0m   ✅ Network & DNS configuration complete
[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
  10.100.1.214    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com


[0;32m[OK][0m   PostgreSQL 'identity-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key \
        host=db-identity-sau-main-dev-postgresql-worker-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[0;34m[INFO][0m Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-worker-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.214
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-worker-01
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 2a8d7237-0c1b-4286-8ffc-cd46f4f7052e
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[0;32m[OK][0m   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 📦 03 role...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 10:53:52 UTC] USER=www-data EUID=0 PID=1949184 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    debezium_user
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   debezium_user
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 10:54:19 UTC] USER=www-data EUID=0 PID=1949437 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-debezium_user
[2026-01-02 10:54:19 UTC] USER=www-data EUID=0 PID=1949446 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-01-02 10:54:19 UTC] USER=www-data EUID=0 PID=1949455 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-debezium_user/ra_root.key
[2026-01-02 10:54:19 UTC] USER=www-data EUID=0 PID=1949464 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-01-02 10:54:19 UTC] USER=www-data EUID=0 PID=1949473 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949489 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949498 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949507 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949516 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949525 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949534 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949543 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949552 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949561 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949570 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949579 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949588 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949597 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949606 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949615 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949624 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949652 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949661 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:20 UTC] USER=www-data EUID=0 PID=1949671 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949689 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949698 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949707 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949716 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949726 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949736 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949745 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949756 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949766 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949775 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949784 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949793 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949802 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949811 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949820 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949830 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949839 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949848 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 10:54:21 UTC] USER=www-data EUID=0 PID=1949857 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1949867 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1949878 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1949887 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1949896 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1949905 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1949914 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1949923 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1949932 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1949942 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1949951 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1949960 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1949969 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1949979 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1949990 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1949999 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1950008 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1950017 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1950026 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1950035 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1950044 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1950053 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:22 UTC] USER=www-data EUID=0 PID=1950062 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:54:23 UTC] USER=www-data EUID=0 PID=1950071 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 10:54:23 UTC] USER=www-data EUID=0 PID=1950080 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-02 10:54:23 UTC] USER=www-data EUID=0 PID=1950090 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:54:23 UTC] USER=www-data EUID=0 PID=1950100 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:54:23 UTC] USER=www-data EUID=0 PID=1950109 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:54:23 UTC] USER=www-data EUID=0 PID=1950118 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 10:54:23 UTC] USER=www-data EUID=0 PID=1950130 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 10:54:23 UTC] USER=www-data EUID=0 PID=1950142 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 10:54:23 UTC] USER=www-data EUID=0 PID=1950151 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:54:23 UTC] USER=www-data EUID=0 PID=1950166 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:54:23 UTC] USER=www-data EUID=0 PID=1950176 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:54:23 UTC] USER=www-data EUID=0 PID=1950185 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: debezium_user
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U debezium_user -d postgres

🔐 Generating replicator client certificate for worker-01...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 10:54:24 UTC] USER=www-data EUID=0 PID=1950228 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-02 10:54:24 UTC] USER=www-data EUID=0 PID=1950237 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 10:54:24 UTC] USER=www-data EUID=0 PID=1950246 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-02 10:54:24 UTC] USER=www-data EUID=0 PID=1950255 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 10:54:24 UTC] USER=www-data EUID=0 PID=1950264 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:54:25 UTC] USER=www-data EUID=0 PID=1950289 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:54:25 UTC] USER=www-data EUID=0 PID=1950300 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:54:25 UTC] USER=www-data EUID=0 PID=1950309 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:54:25 UTC] USER=www-data EUID=0 PID=1950318 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 10:54:25 UTC] USER=www-data EUID=0 PID=1950337 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:25 UTC] USER=www-data EUID=0 PID=1950351 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:54:25 UTC] USER=www-data EUID=0 PID=1950366 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:54:25 UTC] USER=www-data EUID=0 PID=1950381 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:54:25 UTC] USER=www-data EUID=0 PID=1950426 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:54:25 UTC] USER=www-data EUID=0 PID=1950489 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:54:25 UTC] USER=www-data EUID=0 PID=1950580 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:54:25 UTC] USER=www-data EUID=0 PID=1950702 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:25 UTC] USER=www-data EUID=0 PID=1950723 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950732 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950744 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950753 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950763 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950772 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950798 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950807 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950816 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950825 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950834 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950843 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950852 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950861 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950872 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950881 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950890 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950900 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950910 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950919 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950928 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:26 UTC] USER=www-data EUID=0 PID=1950937 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1950946 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1950955 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1950964 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1950973 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1950982 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1950991 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951001 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951012 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951022 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951031 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951040 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951049 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951058 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951067 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951076 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951085 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951094 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951103 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951113 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951124 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951134 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951143 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951152 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951161 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951170 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:27 UTC] USER=www-data EUID=0 PID=1951179 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:54:28 UTC] USER=www-data EUID=0 PID=1951188 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 10:54:28 UTC] USER=www-data EUID=0 PID=1951197 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:28 UTC] USER=www-data EUID=0 PID=1951206 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:54:28 UTC] USER=www-data EUID=0 PID=1951216 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:54:28 UTC] USER=www-data EUID=0 PID=1951225 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:54:28 UTC] USER=www-data EUID=0 PID=1951235 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:54:28 UTC] USER=www-data EUID=0 PID=1951245 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:54:28 UTC] USER=www-data EUID=0 PID=1951254 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:54:28 UTC] USER=www-data EUID=0 PID=1951263 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 10:54:28 UTC] USER=www-data EUID=0 PID=1951274 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 10:54:28 UTC] USER=www-data EUID=0 PID=1951285 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 10:54:28 UTC] USER=www-data EUID=0 PID=1951294 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:54:28 UTC] USER=www-data EUID=0 PID=1951303 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:54:28 UTC] USER=www-data EUID=0 PID=1951312 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:54:28 UTC] USER=www-data EUID=0 PID=1951321 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

✅ Replicator certificate generated for worker-01
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    identity[0m
[0;34m  Zone:       sau[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: worker-01[0m

[0;34mAWS Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-identity-sau-main-dev-worker-01[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mGenerating new secure password...[0m
[0;34mUser fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: fastorder_admin_gd[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials worker-01[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: worker-01/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    fastorder_admin_gd
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   fastorder_admin_gd
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 10:54:51 UTC] USER=www-data EUID=0 PID=1951757 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-fastorder_admin_gd
[2026-01-02 10:54:51 UTC] USER=www-data EUID=0 PID=1951766 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-01-02 10:54:51 UTC] USER=www-data EUID=0 PID=1951775 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
[2026-01-02 10:54:51 UTC] USER=www-data EUID=0 PID=1951784 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-01-02 10:54:51 UTC] USER=www-data EUID=0 PID=1951793 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951808 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951818 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951827 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951836 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951845 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951854 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951864 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951873 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951882 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951891 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951900 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951909 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951918 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951928 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951938 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951947 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951956 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951965 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:52 UTC] USER=www-data EUID=0 PID=1951991 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952000 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952009 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952018 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952029 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952038 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952047 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952056 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952065 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952074 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952083 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952097 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952107 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952116 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952125 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952134 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952143 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952152 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952161 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952170 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952179 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952188 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952197 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952207 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:53 UTC] USER=www-data EUID=0 PID=1952217 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952226 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952235 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952244 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952253 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952262 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952271 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952280 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952289 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952298 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952307 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952317 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952327 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952336 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952345 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952354 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952363 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952372 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952383 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952393 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952402 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952411 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952420 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 10:54:54 UTC] USER=www-data EUID=0 PID=1952430 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:54:55 UTC] USER=www-data EUID=0 PID=1952440 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:54:55 UTC] USER=www-data EUID=0 PID=1952449 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:54:55 UTC] USER=www-data EUID=0 PID=1952458 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 10:54:55 UTC] USER=www-data EUID=0 PID=1952467 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 10:54:55 UTC] USER=www-data EUID=0 PID=1952478 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 10:54:55 UTC] USER=www-data EUID=0 PID=1952488 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:54:55 UTC] USER=www-data EUID=0 PID=1952497 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:54:55 UTC] USER=www-data EUID=0 PID=1952507 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:54:55 UTC] USER=www-data EUID=0 PID=1952516 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: fastorder_admin_gd
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-identity-sau-main-dev-worker-01:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_identity_sau_main_dev_db already exists, skipping creation
[2026-01-02 10:54:56 UTC] USER=www-data EUID=0 PID=1952574 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-02 10:54:56 UTC] USER=www-data EUID=0 PID=1952608 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

[0;34m=== Pre-flight Checks ===[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;32m✓[0m AWS Secrets Manager accessible

[0;34m=== Retrieving Credentials from AWS ===[0m
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
/opt/fastorder/bash/infra_core/cache.sh: line 145: /var/cache/secrets/fastorder_db_identity_sau_main_dev_postgresql_worker-01_fastorder_admin_gd.cache.tmp.1952632: Permission denied
✅ Retrieved from secrets manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials loaded for worker-01/fastorder_admin_gd: fastorder_admin_gd@db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_identity_sau_main_dev_db
[0;32m✓[0m Credentials retrieved: fastorder_admin_gd@db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_identity_sau_main_dev_db
[0;34m╔════════════════════════════════════════════╗[0m
[0;34m║  PostgreSQL Test Suite (AWS Secrets MGR)  ║[0m
[0;34m╚════════════════════════════════════════════╝[0m

[0;34m=== PostgreSQL Authentication Test ===[0m
[0;31m✗[0m PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-identity-sau-main-dev-postgresql-worker-01.fastorder.com" (10.100.1.214), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 10:55:09 UTC] USER=www-data EUID=0 PID=1952894 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_identity_sau_main_dev_db
  SCHEMA      : auth
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
  DNS → 10.100.1.214
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    identity[0m
[0;34m  Zone:       sau[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: worker-01[0m

[0;34mAWS Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-identity-sau-main-dev-worker-01[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mGenerating new secure password...[0m
[0;34mUser debezium_user does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: debezium_user[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials worker-01[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: worker-01/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-01-02 10:55:29 UTC] USER=www-data EUID=0 PID=1953591 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
[2026-01-02 10:55:29 UTC] USER=www-data EUID=0 PID=1953619 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d fastorder_identity_sau_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
🔐 Creating replicator role for worker-01...
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
  DNS → 10.100.1.214
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Creating role: replicator with password
SET
CREATE ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/identity/sau/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🎉 Done.
✅ Replicator role created for worker-01

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=worker-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)

[0;35m╔════════════════════════════════════════════════════════════════════════════╗[0m
[0;35m║                    IAM Database Schema Initialization                       ║[0m
[0;35m╚════════════════════════════════════════════════════════════════════════════╝[0m

[0;34m[INFO][0m 🟢 Starting IAM schema provisioning...
[0;34m[INFO][0m Environment: identity-sau-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m VM IP: 142.93.238.16

[0;34m[INFO][0m 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;34m  Schema: core[0m
[0;34m  Core Identity Directory (tenants, realms, identities, devices, MFA)[0m
[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m

[0;34m[INFO][0m 🔸 Table [1/20]: core/01-tenant
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "pgcrypto" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "citext" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
NOTICE:  schema "utils" already exists, skipping
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
NOTICE:  schema "core" already exists, skipping
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
NOTICE:  relation "tenant" already exists, skipping
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[0;32m[OK][0m Table core/01-tenant initialized

[0;34m[INFO][0m 🔸 Table [2/20]: core/02-realm
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  relation "realm" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_realm_keycloak_id" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_realm_tenant" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[0;32m[OK][0m Table core/02-realm initialized

[0;34m[INFO][0m 🔸 Table [3/20]: core/03-identity
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  relation "identity" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_unique_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_unique_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_type" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[0;32m[OK][0m Table core/03-identity initialized

[0;34m[INFO][0m 🔸 Table [4/20]: core/04-device
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  relation "device" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_device_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_fingerprint" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_trusted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_last_seen" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[0;32m[OK][0m Table core/04-device initialized

[0;34m[INFO][0m 🔸 Table [5/20]: core/05-identity_account
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  relation "identity_account" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_account_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_lockout" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_last_login" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[0;32m[OK][0m Table core/05-identity_account initialized

[0;34m[INFO][0m 🔸 Table [6/20]: core/06-identity_mfa
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  relation "identity_mfa" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_mfa_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_active" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[0;32m[OK][0m Table core/06-identity_mfa initialized

[0;34m[INFO][0m 🔸 Table [7/20]: core/07-external_idp_link
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  relation "external_idp_link" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_external_idp_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_provider" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_email" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[0;32m[OK][0m Table core/07-external_idp_link initialized


[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;34m  Schema: policy[0m
[0;34m  RBAC/ABAC Authorization (clients, roles, permissions, policies)[0m
[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m

[0;34m[INFO][0m 🔸 Table [8/20]: policy/01-client
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
NOTICE:  schema "policy" already exists, skipping
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  relation "client" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_client_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_key" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_status" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[0;32m[OK][0m Table policy/01-client initialized

[0;34m[INFO][0m 🔸 Table [9/20]: policy/02-resource
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  relation "resource" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_resource_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_external" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_owner" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[0;32m[OK][0m Table policy/02-resource initialized

[0;34m[INFO][0m 🔸 Table [10/20]: policy/03-scope
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  relation "scope" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_scope_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_scope_name" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[0;32m[OK][0m Table policy/03-scope initialized

[0;34m[INFO][0m 🔸 Table [11/20]: policy/04-permission
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  relation "permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_permission_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_resource" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[0;32m[OK][0m Table policy/04-permission initialized

[0;34m[INFO][0m 🔸 Table [12/20]: policy/05-role
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  relation "role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_keycloak" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[0;32m[OK][0m Table policy/05-role initialized

[0;34m[INFO][0m 🔸 Table [13/20]: policy/06-role_permission
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  relation "role_permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_permission_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_permission_perm" already exists, skipping
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[0;32m[OK][0m Table policy/06-role_permission initialized

[0;34m[INFO][0m 🔸 Table [14/20]: policy/07-identity_role
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  relation "identity_role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_role_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_active" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[0;32m[OK][0m Table policy/07-identity_role initialized

[0;34m[INFO][0m 🔸 Table [15/20]: policy/08-policy_rule
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  relation "policy_rule" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_policy_rule_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_enabled" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_priority" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[0;32m[OK][0m Table policy/08-policy_rule initialized

[0;34m[INFO][0m 🔸 Table [16/20]: policy/09-api_key
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  relation "api_key" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_api_key_prefix" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[0;32m[OK][0m Table policy/09-api_key initialized


[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;34m  Schema: audit[0m
[0;34m  Audit & Risk Logging (auth events, admin actions, risk decisions)[0m
[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m

[0;34m[INFO][0m 🔸 Table [17/20]: audit/01-auth_event
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
NOTICE:  schema "audit" already exists, skipping
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
NOTICE:  relation "auth_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_auth_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_result" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_ip" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_session" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_trace" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_risk" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[0;32m[OK][0m Table audit/01-auth_event initialized

[0;34m[INFO][0m 🔸 Table [18/20]: audit/02-admin_action
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
NOTICE:  relation "admin_action" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_admin_action_actor" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_target" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_trace" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[0;32m[OK][0m Table audit/02-admin_action initialized

[0;34m[INFO][0m 🔸 Table [19/20]: audit/03-risk_decision
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
NOTICE:  relation "risk_decision" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_risk_decision_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_level" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_decision" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_auth" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[0;32m[OK][0m Table audit/03-risk_decision initialized

[0;34m[INFO][0m 🔸 Table [20/20]: audit/04-consent_event
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
NOTICE:  relation "consent_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_consent_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_version" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_granted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  relation "audit.auth_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  relation "audit.auth_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  relation "audit.admin_action_2026_03" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  relation "audit.admin_action_2026_04" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  relation "audit.risk_decision_2026_03" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  relation "audit.risk_decision_2026_04" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  relation "audit.consent_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  relation "audit.consent_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[0;32m[OK][0m Table audit/04-consent_event initialized


[0;35m════════════════════════════════════════════════════════════════════════════[0m
[0;32m[OK][0m ✅ IAM Schema Initialization Complete!
[0;32m[OK][0m All 20 tables initialized successfully

[0;34mSchemas created:[0m
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

[0;34mDesign highlights:[0m
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

[0;35m════════════════════════════════════════════════════════════════════════════[0m

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=worker-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: identity/run.sh started for SERVICE=identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity
🔍 DEBUG_CHECKPOINT_A3: SERVICE_ROOT exists, discovering table folders
🔍 DEBUG_CHECKPOINT_A4: Found subfolder: auth
🔍 DEBUG_CHECKPOINT_A4b: Checking for nested schema layout in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth
🔍 DEBUG_CHECKPOINT_A4c: Found nested steps dir: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps (display: auth/login)
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: auth/login|/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
[INFO] 📚 Detected grouped table folders under identity/: auth/login

🔍 DEBUG_CHECKPOINT_A7: Current IDENTIFIER=coordinator
🔍 DEBUG_CHECKPOINT_A8_PROCEED: Processing tables on coordinator/main node
🔍 DEBUG_CHECKPOINT_A9: Processing table: auth/login at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
[INFO] 🔸 Table group: auth/login
🔍 DEBUG_CHECKPOINT_A10: About to run numbered steps for table: auth/login
🔍 DEBUG_CHECKPOINT_B1: run_all_numbered_steps_in_dir called for dir=/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps table=auth/login
🔍 DEBUG_CHECKPOINT_B2: Found 1 numbered steps: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B3: About to run step: 01-init-schema.sh
Ab substep 0 compelete start
[DEBUG] Tracking substep start: steps/01-install/steps/identity/auth/login/01-init-schema (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
Ab substep 0 compelete start
[INFO] 📦 01 init schema...
Ab substep 1 compelete start
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing auth.login_account table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating auth schema...
NOTICE:  schema "auth" already exists, skipping
CREATE SCHEMA
✅ Schema created
🔧 Creating account_status ENUM...
DO
✅ ENUM created
🔧 Creating auth.login_account table...
NOTICE:  relation "login_account" already exists, skipping
CREATE TABLE
✅ Table created (Citus-compatible with region_hint in all constraints)
🔧 Creating indexes...
NOTICE:  relation "idx_login_account_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_login_account_username" already exists, skipping
CREATE INDEX
✅ Indexes created
ℹ️  Table already registered with Citus
🎉 Schema initialization complete for fastorder_identity_sau_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

📊 Registering environment in monitoring database (obs schema)...
   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
   Resource IP: 142.93.238.16
⚠️  Could not connect to monitoring database, skipping registration
   You can manually register later using:
   /opt/fastorder/bash/scripts/env_app_setup/setup/04-postgresql/steps/register-authN-af-aaaa1-dev.sh

==========================================
✅ Schema initialization complete!
==========================================
Ab substep 1 compelete end
Ab substep 2 compelete start
Ab substep 2 compelete end

🔍 DEBUG_CHECKPOINT_B4: Completed step: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B5: All numbered steps completed for /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A11: Completed numbered steps for table: auth/login
compeleted here

🔍 DEBUG_CHECKPOINT_A12: All tables processed
End of 04-postgresql/steps/01-install/steps/identity/run.sh

[0;32m✓[0m ✅ Worker worker-01 setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up standby replicas (1 per worker)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up standby: worker-01-standby-01 (replica of worker-01)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-02 10:57:37 UTC] USER=unknown EUID=33 PID=1957516 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 10:57:38 UTC] USER=unknown EUID=33 PID=1957525 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 10:57:38 UTC] USER=unknown EUID=33 PID=1957533 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-02 10:57:38 UTC] USER=unknown EUID=33 PID=1957540 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 10:57:38 UTC] USER=unknown EUID=33 PID=1957547 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-02 10:57:38 UTC] USER=unknown EUID=33 PID=1957554 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede JOB_UUID=333bba61-dfcf-4fcb-986b-f0541277e251

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: identity-sau-main-dev
[0;34m[INFO][0m Identifier: worker-01-standby-01
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.211
[0;34m[INFO][0m Primary hostname: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

[0;34m[INFO][0m Adding /etc/hosts entry for worker-01-standby-01...
[0;34m[INFO][0m   db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.211

[0;34m[INFO][0m   ✅ db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com already exists with correct IP

[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;32m✅[0m   ✅ Network & DNS configuration complete
[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
  10.100.1.211    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  identity-sau-main-dev.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 10:57:41 UTC] USER=www-data EUID=0 PID=1957682 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:41 UTC] USER=www-data EUID=0 PID=1957691 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-02 10:57:41 UTC] USER=www-data EUID=0 PID=1957701 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1957649
[2026-01-02 10:57:41 UTC] USER=www-data EUID=0 PID=1957710 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1957649/ra_root.crt
[2026-01-02 10:57:41 UTC] USER=www-data EUID=0 PID=1957719 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1957649/ra_root.key
[2026-01-02 10:57:42 UTC] USER=www-data EUID=0 PID=1957728 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1957649/ra_root.crt
[2026-01-02 10:57:42 UTC] USER=www-data EUID=0 PID=1957737 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1957649/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
[2026-01-02 10:57:47 UTC] USER=www-data EUID=0 PID=1957800 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1957649/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 10:57:47 UTC] USER=www-data EUID=0 PID=1957809 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1957649/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 10:57:47 UTC] USER=www-data EUID=0 PID=1957818 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 10:57:47 UTC] USER=www-data EUID=0 PID=1957827 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1957649/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:57:47 UTC] USER=www-data EUID=0 PID=1957837 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:57:47 UTC] USER=www-data EUID=0 PID=1957846 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:57:47 UTC] USER=www-data EUID=0 PID=1957855 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 10:57:47 UTC] USER=www-data EUID=0 PID=1957866 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 10:57:47 UTC] USER=www-data EUID=0 PID=1957875 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 10:57:47 UTC] USER=www-data EUID=0 PID=1957884 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 10:57:47 UTC] USER=www-data EUID=0 PID=1957893 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 10:57:47 UTC] USER=www-data EUID=0 PID=1957902 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:47 UTC] USER=www-data EUID=0 PID=1957911 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:identity-sau-main-dev.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01-standby-01
Primary CN:  identity-sau-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=identity-sau-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 10:57:48 UTC] USER=www-data EUID=0 PID=1957966 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-01-02 10:57:48 UTC] USER=www-data EUID=0 PID=1957975 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 10:57:48 UTC] USER=www-data EUID=0 PID=1957984 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
[2026-01-02 10:57:48 UTC] USER=www-data EUID=0 PID=1957993 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 10:57:48 UTC] USER=www-data EUID=0 PID=1958002 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:48 UTC] USER=www-data EUID=0 PID=1958022 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:48 UTC] USER=www-data EUID=0 PID=1958031 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:48 UTC] USER=www-data EUID=0 PID=1958040 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 10:57:48 UTC] USER=www-data EUID=0 PID=1958049 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 10:57:48 UTC] USER=www-data EUID=0 PID=1958059 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:57:48 UTC] USER=www-data EUID=0 PID=1958069 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958087 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958096 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958105 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958114 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958123 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958132 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958141 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958150 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958159 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958185 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958194 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958204 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958213 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958222 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958231 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958240 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958249 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958258 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958267 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958276 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958286 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958296 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958305 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:49 UTC] USER=www-data EUID=0 PID=1958315 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958324 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958333 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958342 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958351 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958360 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958369 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958378 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958387 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958397 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958407 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958416 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958425 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958434 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958446 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958455 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958464 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958473 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958482 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958491 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958500 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958510 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958520 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958529 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:50 UTC] USER=www-data EUID=0 PID=1958540 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:51 UTC] USER=www-data EUID=0 PID=1958550 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:51 UTC] USER=www-data EUID=0 PID=1958560 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:51 UTC] USER=www-data EUID=0 PID=1958569 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 10:57:51 UTC] USER=www-data EUID=0 PID=1958578 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 10:57:51 UTC] USER=www-data EUID=0 PID=1958587 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:57:51 UTC] USER=www-data EUID=0 PID=1958596 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 10:57:51 UTC] USER=www-data EUID=0 PID=1958605 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 10:57:51 UTC] USER=www-data EUID=0 PID=1958614 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 10:57:51 UTC] USER=www-data EUID=0 PID=1958624 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:57:51 UTC] USER=www-data EUID=0 PID=1958634 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:57:51 UTC] USER=www-data EUID=0 PID=1958643 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:57:51 UTC] USER=www-data EUID=0 PID=1958654 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 10:57:51 UTC] USER=www-data EUID=0 PID=1958663 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 10:57:51 UTC] USER=www-data EUID=0 PID=1958672 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 10:57:51 UTC] USER=www-data EUID=0 PID=1958681 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:57:51 UTC] USER=www-data EUID=0 PID=1958699 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:57:51 UTC] USER=www-data EUID=0 PID=1958708 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 10:57:52 UTC] USER=www-data EUID=0 PID=1958751 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-01-02 10:57:52 UTC] USER=www-data EUID=0 PID=1958760 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 10:57:52 UTC] USER=www-data EUID=0 PID=1958769 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
[2026-01-02 10:57:52 UTC] USER=www-data EUID=0 PID=1958778 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 10:57:52 UTC] USER=www-data EUID=0 PID=1958788 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:52 UTC] USER=www-data EUID=0 PID=1958802 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:52 UTC] USER=www-data EUID=0 PID=1958811 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:52 UTC] USER=www-data EUID=0 PID=1958820 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 10:57:52 UTC] USER=www-data EUID=0 PID=1958829 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 10:57:52 UTC] USER=www-data EUID=0 PID=1958838 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:57:52 UTC] USER=www-data EUID=0 PID=1958847 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 10:57:52 UTC] USER=www-data EUID=0 PID=1958856 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1958865 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1958874 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1958883 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1958892 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1958901 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1958910 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1958919 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1958928 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1958937 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1958946 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1958955 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1958987 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1958996 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1959005 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1959014 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1959023 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1959032 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1959041 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1959050 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1959059 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1959068 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1959077 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 10:57:53 UTC] USER=www-data EUID=0 PID=1959089 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959099 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959108 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959117 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959126 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959135 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959144 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959153 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959162 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959171 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959180 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959189 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959199 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959209 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959219 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959228 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959237 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959246 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959255 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959264 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959273 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959282 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959291 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959300 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959310 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959320 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959330 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:54 UTC] USER=www-data EUID=0 PID=1959339 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959348 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959357 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959366 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959375 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959384 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959393 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959402 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959411 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959421 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959431 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959440 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959449 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959459 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959470 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959479 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959488 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959497 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:57:55 UTC] USER=www-data EUID=0 PID=1959506 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 10:57:56 UTC] USER=www-data EUID=0 PID=1959549 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-02 10:57:56 UTC] USER=www-data EUID=0 PID=1959558 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 10:57:56 UTC] USER=www-data EUID=0 PID=1959567 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-02 10:57:56 UTC] USER=www-data EUID=0 PID=1959576 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 10:57:56 UTC] USER=www-data EUID=0 PID=1959585 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:57:56 UTC] USER=www-data EUID=0 PID=1959600 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:57:56 UTC] USER=www-data EUID=0 PID=1959609 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:57:56 UTC] USER=www-data EUID=0 PID=1959618 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:57:56 UTC] USER=www-data EUID=0 PID=1959627 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959645 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959654 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959663 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959672 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959681 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959690 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959699 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959708 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959717 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959726 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959735 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959744 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959782 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959800 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959809 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959818 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959827 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959836 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959845 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959856 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:57:57 UTC] USER=www-data EUID=0 PID=1959866 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:57:58 UTC] USER=www-data EUID=0 PID=1959875 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:57:58 UTC] USER=www-data EUID=0 PID=1959885 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:57:58 UTC] USER=www-data EUID=0 PID=1959895 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:57:58 UTC] USER=www-data EUID=0 PID=1959904 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:58 UTC] USER=www-data EUID=0 PID=1959914 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:58 UTC] USER=www-data EUID=0 PID=1959923 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:57:58 UTC] USER=www-data EUID=0 PID=1959932 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:57:58 UTC] USER=www-data EUID=0 PID=1959943 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:57:58 UTC] USER=www-data EUID=0 PID=1959952 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 10:57:58 UTC] USER=www-data EUID=0 PID=1959965 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:57:58 UTC] USER=www-data EUID=0 PID=1959974 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:57:58 UTC] USER=www-data EUID=0 PID=1959983 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:57:58 UTC] USER=www-data EUID=0 PID=1959992 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:57:58 UTC] USER=www-data EUID=0 PID=1960002 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:57:58 UTC] USER=www-data EUID=0 PID=1960030 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:58 UTC] USER=www-data EUID=0 PID=1960039 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:57:59 UTC] USER=www-data EUID=0 PID=1960048 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:57:59 UTC] USER=www-data EUID=0 PID=1960057 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:57:59 UTC] USER=www-data EUID=0 PID=1960066 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 10:57:59 UTC] USER=www-data EUID=0 PID=1960075 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:57:59 UTC] USER=www-data EUID=0 PID=1960084 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:57:59 UTC] USER=www-data EUID=0 PID=1960093 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:57:59 UTC] USER=www-data EUID=0 PID=1960102 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:57:59 UTC] USER=www-data EUID=0 PID=1960112 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:57:59 UTC] USER=www-data EUID=0 PID=1960122 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:57:59 UTC] USER=www-data EUID=0 PID=1960131 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:57:59 UTC] USER=www-data EUID=0 PID=1960151 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:57:59 UTC] USER=www-data EUID=0 PID=1960160 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:57:59 UTC] USER=www-data EUID=0 PID=1960169 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:57:59 UTC] USER=www-data EUID=0 PID=1960178 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 10:57:59 UTC] USER=www-data EUID=0 PID=1960187 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:57:59 UTC] USER=www-data EUID=0 PID=1960196 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:57:59 UTC] USER=www-data EUID=0 PID=1960205 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:58:00 UTC] USER=www-data EUID=0 PID=1960214 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:58:00 UTC] USER=www-data EUID=0 PID=1960224 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:58:00 UTC] USER=www-data EUID=0 PID=1960234 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:58:00 UTC] USER=www-data EUID=0 PID=1960245 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:58:00 UTC] USER=www-data EUID=0 PID=1960254 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 10:58:00 UTC] USER=www-data EUID=0 PID=1960263 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 10:58:00 UTC] USER=www-data EUID=0 PID=1960272 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 10:58:00 UTC] USER=www-data EUID=0 PID=1960281 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:58:00 UTC] USER=www-data EUID=0 PID=1960290 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:58:00 UTC] USER=www-data EUID=0 PID=1960299 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:58:00 UTC] USER=www-data EUID=0 PID=1960308 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 📦 02 setup pg instance...
[0;34m[DEADLOCK-PREVENTION][0m Deadlock prevention library loaded
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-standby-01-postgresql environment: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com (10.100.1.211)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.211
[0;34m[INFO][0m Environment: identity-sau-main-dev
[0;34m[INFO][0m Identifier: worker-01-standby-01
[0;34m[INFO][0m Data dir:   /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[0;34m[INFO][0m Port:       5432
[0;34m[INFO][0m Hostname:   db-identity-sau-main-dev-postgresql-worker-01-standby-01
[2026-01-02 10:58:03 UTC] USER=www-data EUID=0 PID=1960431 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:58:03 UTC] USER=www-data EUID=0 PID=1960454 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:58:03 UTC] USER=www-data EUID=0 PID=1960479 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:58:03 UTC] USER=www-data EUID=0 PID=1960500 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[1;33m[WARN][0m Server certificate not found at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[0;34m[INFO][0m Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  identity-sau-main-dev.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 10:58:04 UTC] USER=www-data EUID=0 PID=1960540 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:58:04 UTC] USER=www-data EUID=0 PID=1960549 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-02 10:58:04 UTC] USER=www-data EUID=0 PID=1960559 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1960507
[2026-01-02 10:58:04 UTC] USER=www-data EUID=0 PID=1960568 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1960507/ra_root.crt
[2026-01-02 10:58:04 UTC] USER=www-data EUID=0 PID=1960577 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1960507/ra_root.key
[2026-01-02 10:58:04 UTC] USER=www-data EUID=0 PID=1960586 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1960507/ra_root.crt
[2026-01-02 10:58:04 UTC] USER=www-data EUID=0 PID=1960595 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1960507/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
[2026-01-02 10:58:06 UTC] USER=www-data EUID=0 PID=1960635 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1960507/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 10:58:06 UTC] USER=www-data EUID=0 PID=1960644 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1960507/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 10:58:06 UTC] USER=www-data EUID=0 PID=1960653 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 10:58:06 UTC] USER=www-data EUID=0 PID=1960662 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1960507/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:58:06 UTC] USER=www-data EUID=0 PID=1960671 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:58:06 UTC] USER=www-data EUID=0 PID=1960681 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-02 10:58:06 UTC] USER=www-data EUID=0 PID=1960690 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 10:58:06 UTC] USER=www-data EUID=0 PID=1960701 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 10:58:06 UTC] USER=www-data EUID=0 PID=1960710 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 10:58:06 UTC] USER=www-data EUID=0 PID=1960719 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 10:58:06 UTC] USER=www-data EUID=0 PID=1960728 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 10:58:06 UTC] USER=www-data EUID=0 PID=1960737 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:58:06 UTC] USER=www-data EUID=0 PID=1960746 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:identity-sau-main-dev.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01-standby-01
Primary CN:  identity-sau-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=identity-sau-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-02 10:58:06 UTC] USER=www-data EUID=0 PID=1960775 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-02 10:58:06 UTC] USER=www-data EUID=0 PID=1960784 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-02 10:58:06 UTC] USER=www-data EUID=0 PID=1960793 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[0;32m[OK][0m   mTLS certificates OK (server cert + client certs verified) and keys secured
[0;34m[INFO][0m Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-02 10:58:06 UTC] USER=www-data EUID=0 PID=1960816 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-worker-01-standby-01.service
[2026-01-02 10:58:07 UTC] USER=www-data EUID=0 PID=1960842 ACTION=passthru ARGS=systemctl stop postgresql
[1;33m[WARN][0m Cleaning stale socket directory /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[2026-01-02 10:58:07 UTC] USER=www-data EUID=0 PID=1960873 ACTION=fsop ARGS=rm -rf /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[0;32m[OK][0m   No conflicting Postgres left on port 5432
[0;32m[OK][0m   Generated new postgres password for initdb
[2026-01-02 10:58:37 UTC] USER=www-data EUID=0 PID=1961403 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.LYb4DZ
[2026-01-02 10:58:37 UTC] USER=www-data EUID=0 PID=1961426 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.LYb4DZ
[2026-01-02 10:58:37 UTC] USER=www-data EUID=0 PID=1961449 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 10:58:37 UTC] USER=www-data EUID=0 PID=1961471 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-02 10:58:38 UTC] USER=www-data EUID=0 PID=1961495 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/identity-sau-main-dev
[0;34m[INFO][0m This is a standby. Using pg_basebackup from primary (worker-01)...
[0;34m[INFO][0m Setting up replicator role and slot on primary (worker-01)...
ℹ️  Scanning primary for stuck queries from previous failed attempts...
ℹ️  Scanning for stuck queries (timeout: 30s)...
ℹ️  No stuck queries found
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SLOT        : worker_01_standby_01
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
  DNS → 10.100.1.214
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Role replicator already exists, updating password and ensuring REPLICATION privilege
SET
ALTER ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/identity/sau/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🔧 Ensuring replication slot: worker_01_standby_01…
🆕 Creating replication slot worker_01_standby_01
SET
 pg_create_physical_replication_slot 
-------------------------------------
 (worker_01_standby_01,)
(1 row)

✅ Replication slot worker_01_standby_01 created.
🎉 Done.
[0;32m[OK][0m   Replicator role and slot created on primary
[0;34m[INFO][0m Creating replicator client certificates for connecting to primary (worker-01)...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 10:58:44 UTC] USER=www-data EUID=0 PID=1961648 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-02 10:58:44 UTC] USER=www-data EUID=0 PID=1961658 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 10:58:44 UTC] USER=www-data EUID=0 PID=1961667 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-02 10:58:44 UTC] USER=www-data EUID=0 PID=1961677 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 10:58:44 UTC] USER=www-data EUID=0 PID=1961686 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:58:44 UTC] USER=www-data EUID=0 PID=1961700 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:58:44 UTC] USER=www-data EUID=0 PID=1961709 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961718 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961727 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961736 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961745 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961757 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961766 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961775 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961793 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961802 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961811 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961820 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961829 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961838 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961847 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961857 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961885 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961894 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961903 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961912 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961921 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961930 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961939 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961948 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:58:45 UTC] USER=www-data EUID=0 PID=1961957 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1961966 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1961975 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1961985 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1961995 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962004 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962013 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962022 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962031 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962040 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962049 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962058 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962067 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962076 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962085 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962095 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962105 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962114 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962123 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962132 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962141 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962150 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962159 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962168 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962177 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962188 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:58:46 UTC] USER=www-data EUID=0 PID=1962197 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962207 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962219 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962228 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962237 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962246 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962255 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962264 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962273 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962282 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962291 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962302 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962311 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962321 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962331 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962340 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962349 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962358 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962367 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962376 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962385 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962394 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962403 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

[0;32m[OK][0m   Replicator certificate created for worker-01 in /home/postgres/
[0;34m[INFO][0m Using replicator certificates from primary worker-01...
[2026-01-02 10:58:47 UTC] USER=www-data EUID=0 PID=1962432 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 10:58:48 UTC] USER=www-data EUID=0 PID=1962453 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-02 10:58:48 UTC] USER=www-data EUID=0 PID=1962474 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[0;32m[OK][0m   Replicator certificates verified at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[0;32m[OK][0m   root.crt verified at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[0;34m[INFO][0m Updating primary pg_hba.conf to allow replication...
[0;34m[INFO][0m   Standby IP: 10.100.1.211/32 (standby's source IP)
[0;34m[INFO][0m   Primary application IP: 10.100.1.214/32 (for local pg_basebackup)
[0;34m[INFO][0m   Primary DNS IP: 10.100.1.214/32 (DNS resolution of db-identity-sau-main-dev-postgresql-worker-01.fastorder.com)
[2026-01-02 10:58:48 UTC] USER=www-data EUID=0 PID=1962505 ACTION=passthru ARGS=grep -qxF # BEGIN standby-replication (managed) /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 10:58:48 UTC] USER=www-data EUID=0 PID=1962552 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.211/32  scram-sha-256 
      $0==begin {inside=1}
      inside && $0==rule {found=1}
      $0==end {inside=0}
      END {exit found?0:1}
     /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 10:58:48 UTC] USER=www-data EUID=0 PID=1962576 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl  replication  replicator  10.100.1.211/32  scram-sha-256 /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-02 10:58:48 UTC] USER=www-data EUID=0 PID=1962598 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.214/32  scram-sha-256 
        $0==begin {inside=1}
        inside && $0==rule {found=1}
        $0==end {inside=0}
        END {exit found?0:1}
       /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[0;34m[INFO][0m Reloading primary PostgreSQL service...
[2026-01-02 10:58:49 UTC] USER=www-data EUID=0 PID=1962644 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
[0;32m[OK][0m   Primary pg_hba.conf updated and service reloaded
[1;33m[WARN][0m Removing existing data directory: /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[2026-01-02 10:58:49 UTC] USER=www-data EUID=0 PID=1962668 ACTION=fsop ARGS=rm -rf /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[0;34m[INFO][0m Primary host: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[0;34m[INFO][0m Using replicator cert: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[0;34m[INFO][0m Using replicator key: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key (PKCS#8 format)
[0;34m[INFO][0m Using CA cert: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[0;34m[INFO][0m Verifying postgres user can access certificates...
[0;31m[ERR][0m  postgres user CANNOT read /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[0;34m[INFO][0m File permissions:
lrwxrwxrwx 1 postgres ssl-cert 72 Jan  2 10:58 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt -> /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[0;34m[INFO][0m Parent directory permissions:
drwx------ 2 postgres postgres 4096 Jan  2 10:58 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
drwx------ 6 postgres postgres 4096 Jan  2 07:10 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[1;33m[WARN][0m Attempting to fix permissions (/usr/local/bin/fastorder-provisioning-wrapper.sh required)...
[0;34m[INFO][0m Fixing /home/postgres/ directory...
[2026-01-02 10:58:49 UTC] USER=www-data EUID=0 PID=1962735 ACTION=fsop ARGS=chmod 755 /home/postgres/
[0;34m[INFO][0m Fixing /home/postgres/ssl/.postgresql/...
[2026-01-02 10:58:49 UTC] USER=www-data EUID=0 PID=1962756 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/
[0;34m[INFO][0m Fixing parent directory: /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 10:58:50 UTC] USER=www-data EUID=0 PID=1962779 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[0;34m[INFO][0m Fixing certificate directory: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-02 10:58:50 UTC] USER=www-data EUID=0 PID=1962800 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[0;34m[INFO][0m Fixing CA certificate: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-02 10:58:50 UTC] USER=www-data EUID=0 PID=1962821 ACTION=fsop ARGS=chmod 644 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[0;32m[OK][0m   Permissions fixed
[0;32m[OK][0m   postgres user can now read /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt after permission fix
[2026-01-02 10:58:50 UTC] USER=www-data EUID=0 PID=1962842 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[2026-01-02 10:58:50 UTC] USER=www-data EUID=0 PID=1962863 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[2026-01-02 10:58:50 UTC] USER=www-data EUID=0 PID=1962884 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[0;34m[INFO][0m Checking primary database size before pg_basebackup...
[0;34m[INFO][0m Total primary database size: 29 MB
[0;34m[INFO][0m Estimated transfer time: ~0 minutes (at 10MB/s with compression)
[0;34m[INFO][0m Retrieving replicator password from AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/replicator
[0;32m[OK][0m   Replicator password retrieved successfully
[0;34m[INFO][0m Starting pg_basebackup...
[2026-01-02 10:58:54 UTC] USER=www-data EUID=0 PID=1962969 ACTION=passthru ARGS=sudo -u postgres env PGPASSWORD=qrzga0rZrBWHXjHNfE1t9bdwqo6QF84R PGSSLMODE=verify-full PGSSLCERT=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt PGSSLKEY=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key PGSSLROOTCERT=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /usr/lib/postgresql/17/bin/pg_basebackup -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -p 5432 -U replicator -D /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01 -Fp -Xs -P -R --checkpoint=fast --wal-method=stream --verbose
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/2000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1962980"
17638/30540 kB (57%), 0/1 tablespace (...er-01-standby-01/base/16384/2619)
30550/30550 kB (100%), 0/1 tablespace (...-01-standby-01/global/pg_control)
30550/30550 kB (100%), 1/1 tablespace                                         
pg_basebackup: write-ahead log end point: 0/2000120
pg_basebackup: waiting for background process to finish streaming ...
pg_basebackup: syncing data to disk ...
pg_basebackup: renaming backup_manifest.tmp to backup_manifest
pg_basebackup: base backup completed
[0;32m[OK][0m   pg_basebackup complete
[0;34m[INFO][0m Fixing postgresql.auto.conf to use IP-based primary_conninfo (matching golden backup)...
[2026-01-02 10:58:56 UTC] USER=www-data EUID=0 PID=1962997 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-02 10:58:56 UTC] USER=www-data EUID=0 PID=1963019 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-02 10:58:56 UTC] USER=www-data EUID=0 PID=1963040 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-02 10:58:56 UTC] USER=www-data EUID=0 PID=1963049 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[0;32m[OK][0m   standby.signal verified and permissions set
[0;34m[INFO][0m Fixing postgresql.conf with standby-specific settings...
[1;33m[WARN][0m postgresql.conf not found at /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/postgresql.conf
[0;34m[INFO][0m Verifying postgresql.auto.conf...
[1;33m[WARN][0m postgresql.auto.conf not found - pg_basebackup may have failed
[2026-01-02 10:58:56 UTC] USER=www-data EUID=0 PID=1963072 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.LYb4DZ
[0;34m[INFO][0m Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[0;32m[OK][0m   postgresql.conf updated successfully
[0;34m[INFO][0m Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-02 10:58:56 UTC] USER=www-data EUID=0 PID=1963132 ACTION=fsop ARGS=cp /tmp/tmp.TBV6QxsIYy /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/pg_hba.conf
[2026-01-02 10:58:57 UTC] USER=www-data EUID=0 PID=1963153 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/pg_hba.conf
[2026-01-02 10:58:57 UTC] USER=www-data EUID=0 PID=1963174 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/pg_hba.conf
[0;32m[OK][0m   pg_hba.conf updated
[0;34m[INFO][0m Creating systemd unit: /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01-standby-01.service
[2026-01-02 10:58:57 UTC] USER=www-data EUID=0 PID=1963205 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.XbQWF0 /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01-standby-01.service
[2026-01-02 10:58:57 UTC] USER=www-data EUID=0 PID=1963226 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01-standby-01.service
[0;32m[OK][0m   systemd unit written
[2026-01-02 10:58:57 UTC] USER=www-data EUID=0 PID=1963247 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 10:58:57 UTC] USER=www-data EUID=0 PID=1963268 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 10:58:57 UTC] USER=www-data EUID=0 PID=1963289 ACTION=passthru ARGS=systemctl daemon-reload
[0;34m[INFO][0m Starting PostgreSQL instance...
[2026-01-02 10:58:59 UTC] USER=www-data EUID=0 PID=1963412 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-worker-01-standby-01.service
[0;34m[INFO][0m Waiting for ACTIVE (systemd)…
[2026-01-02 10:59:00 UTC] USER=www-data EUID=0 PID=1963454 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-worker-01-standby-01.service
[0;32m[OK][0m   Service ACTIVE
[0;34m[INFO][0m Waiting for port 5432 bind…
[0;32m[OK][0m   Port bound
[0;34m[INFO][0m Waiting pg_isready (socket)…
[0;32m[OK][0m   Readiness via socket OK
[0;34m[INFO][0m Waiting pg_isready (TCP db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com:5432)…
[0;32m[OK][0m   Startup sequence complete
[0;34m[INFO][0m Configuring synchronous replication on primary worker-01...
[0;34m[INFO][0m Current synchronous_standby_names: ''
[0;34m[INFO][0m Initializing synchronous_standby_names with first standby
[0;34m[INFO][0m New synchronous_standby_names: 'ANY 1 (worker_01_standby_01)'
[2026-01-02 10:59:01 UTC] USER=www-data EUID=0 PID=1963536 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_commit = on;
ALTER SYSTEM
[2026-01-02 10:59:01 UTC] USER=www-data EUID=0 PID=1963559 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_standby_names = 'ANY 1 (worker_01_standby_01)';
ALTER SYSTEM
[2026-01-02 10:59:02 UTC] USER=www-data EUID=0 PID=1963582 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[0;32m[OK][0m   ✅ Synchronous replication configured on primary
[0;32m[OK][0m      Setting: ANY 1 (worker_01_standby_01)
[0;34m[INFO][0m Validating core security GUCs (via local socket)…
[0;32m[OK][0m   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[0;34m[INFO][0m Skipping database/role provisioning on standby node (read-only)
[0;34m[INFO][0m   Database/roles will be replicated from primary: worker-01
[0;34m[INFO][0m Applying connection and memory optimizations...
[0;34m[INFO][0m Standby will use primary's max_connections: 100
[0;34m[INFO][0m Current settings: max_connections=100, work_mem=8MB
[0;34m[INFO][0m Target settings (standby): max_connections=100, work_mem=8MB
[0;32m[OK][0m   Connection settings already optimized
[0;34m[INFO][0m Skipping password setting - this is a standby (read-only)
[0;34m[INFO][0m Use primary's postgres password to connect to this standby
[0;34m[INFO][0m Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: identity-sau-main-dev
[0;34m[INFO][0m Identifier: worker-01-standby-01
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.211
[0;34m[INFO][0m Primary hostname: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

[0;34m[INFO][0m Adding /etc/hosts entry for worker-01-standby-01...
[0;34m[INFO][0m   db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.211

[0;34m[INFO][0m   ✅ db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com already exists with correct IP

[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;32m✅[0m   ✅ Network & DNS configuration complete
[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
  10.100.1.211    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com


[0;32m[OK][0m   PostgreSQL 'identity-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key \
        host=db-identity-sau-main-dev-postgresql-worker-01-standby-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[0;34m[INFO][0m Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-worker-01-standby-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.211
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-worker-01-standby-01
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 8eaa8059-bede-4f71-ae1d-d26590a898da
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[0;32m[OK][0m   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 📦 03 role...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 10:59:09 UTC] USER=www-data EUID=0 PID=1963942 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
⚠ This is a PostgreSQL STANDBY (read-only replica)
⚠ Skipping role creation - standby gets roles from primary via replication
⚠ Use the PRIMARY's credentials to connect to this standby


[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=worker-01-standby-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)

[0;35m╔════════════════════════════════════════════════════════════════════════════╗[0m
[0;35m║                    IAM Database Schema Initialization                       ║[0m
[0;35m╚════════════════════════════════════════════════════════════════════════════╝[0m

[0;34m[INFO][0m 🟢 Starting IAM schema provisioning...
[0;34m[INFO][0m Environment: identity-sau-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m VM IP: 142.93.238.16

[0;34m[INFO][0m 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;34m  Schema: core[0m
[0;34m  Core Identity Directory (tenants, realms, identities, devices, MFA)[0m
[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m

[0;34m[INFO][0m 🔸 Table [1/20]: core/01-tenant
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "pgcrypto" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "citext" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
NOTICE:  schema "utils" already exists, skipping
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
NOTICE:  schema "core" already exists, skipping
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
NOTICE:  relation "tenant" already exists, skipping
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[0;32m[OK][0m Table core/01-tenant initialized

[0;34m[INFO][0m 🔸 Table [2/20]: core/02-realm
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  relation "realm" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_realm_keycloak_id" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_realm_tenant" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[0;32m[OK][0m Table core/02-realm initialized

[0;34m[INFO][0m 🔸 Table [3/20]: core/03-identity
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  relation "identity" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_unique_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_unique_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_type" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[0;32m[OK][0m Table core/03-identity initialized

[0;34m[INFO][0m 🔸 Table [4/20]: core/04-device
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  relation "device" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_device_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_fingerprint" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_trusted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_last_seen" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[0;32m[OK][0m Table core/04-device initialized

[0;34m[INFO][0m 🔸 Table [5/20]: core/05-identity_account
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  relation "identity_account" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_account_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_lockout" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_last_login" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[0;32m[OK][0m Table core/05-identity_account initialized

[0;34m[INFO][0m 🔸 Table [6/20]: core/06-identity_mfa
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  relation "identity_mfa" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_mfa_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_active" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[0;32m[OK][0m Table core/06-identity_mfa initialized

[0;34m[INFO][0m 🔸 Table [7/20]: core/07-external_idp_link
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  relation "external_idp_link" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_external_idp_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_provider" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_email" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[0;32m[OK][0m Table core/07-external_idp_link initialized


[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;34m  Schema: policy[0m
[0;34m  RBAC/ABAC Authorization (clients, roles, permissions, policies)[0m
[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m

[0;34m[INFO][0m 🔸 Table [8/20]: policy/01-client
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
NOTICE:  schema "policy" already exists, skipping
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  relation "client" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_client_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_key" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_status" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[0;32m[OK][0m Table policy/01-client initialized

[0;34m[INFO][0m 🔸 Table [9/20]: policy/02-resource
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  relation "resource" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_resource_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_external" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_owner" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[0;32m[OK][0m Table policy/02-resource initialized

[0;34m[INFO][0m 🔸 Table [10/20]: policy/03-scope
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  relation "scope" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_scope_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_scope_name" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[0;32m[OK][0m Table policy/03-scope initialized

[0;34m[INFO][0m 🔸 Table [11/20]: policy/04-permission
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  relation "permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_permission_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_resource" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[0;32m[OK][0m Table policy/04-permission initialized

[0;34m[INFO][0m 🔸 Table [12/20]: policy/05-role
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  relation "role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_keycloak" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[0;32m[OK][0m Table policy/05-role initialized

[0;34m[INFO][0m 🔸 Table [13/20]: policy/06-role_permission
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  relation "role_permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_permission_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_permission_perm" already exists, skipping
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[0;32m[OK][0m Table policy/06-role_permission initialized

[0;34m[INFO][0m 🔸 Table [14/20]: policy/07-identity_role
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  relation "identity_role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_role_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_active" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[0;32m[OK][0m Table policy/07-identity_role initialized

[0;34m[INFO][0m 🔸 Table [15/20]: policy/08-policy_rule
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  relation "policy_rule" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_policy_rule_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_enabled" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_priority" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[0;32m[OK][0m Table policy/08-policy_rule initialized

[0;34m[INFO][0m 🔸 Table [16/20]: policy/09-api_key
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  relation "api_key" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_api_key_prefix" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[0;32m[OK][0m Table policy/09-api_key initialized


[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;34m  Schema: audit[0m
[0;34m  Audit & Risk Logging (auth events, admin actions, risk decisions)[0m
[0;34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m

[0;34m[INFO][0m 🔸 Table [17/20]: audit/01-auth_event
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
NOTICE:  schema "audit" already exists, skipping
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
NOTICE:  relation "auth_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_auth_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_result" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_ip" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_session" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_trace" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_risk" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[0;32m[OK][0m Table audit/01-auth_event initialized

[0;34m[INFO][0m 🔸 Table [18/20]: audit/02-admin_action
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
NOTICE:  relation "admin_action" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_admin_action_actor" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_target" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_trace" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[0;32m[OK][0m Table audit/02-admin_action initialized

[0;34m[INFO][0m 🔸 Table [19/20]: audit/03-risk_decision
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
NOTICE:  relation "risk_decision" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_risk_decision_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_level" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_decision" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_auth" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[0;32m[OK][0m Table audit/03-risk_decision initialized

[0;34m[INFO][0m 🔸 Table [20/20]: audit/04-consent_event
[0;34m[INFO][0m 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
NOTICE:  relation "consent_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_consent_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_version" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_granted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  relation "audit.auth_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  relation "audit.auth_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  relation "audit.admin_action_2026_03" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  relation "audit.admin_action_2026_04" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  relation "audit.risk_decision_2026_03" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  relation "audit.risk_decision_2026_04" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  relation "audit.consent_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  relation "audit.consent_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[0;32m[OK][0m Table audit/04-consent_event initialized


[0;35m════════════════════════════════════════════════════════════════════════════[0m
[0;32m[OK][0m ✅ IAM Schema Initialization Complete!
[0;32m[OK][0m All 20 tables initialized successfully

[0;34mSchemas created:[0m
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

[0;34mDesign highlights:[0m
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

[0;35m════════════════════════════════════════════════════════════════════════════[0m

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=worker-01-standby-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: identity/run.sh started for SERVICE=identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity
🔍 DEBUG_CHECKPOINT_A3: SERVICE_ROOT exists, discovering table folders
🔍 DEBUG_CHECKPOINT_A4: Found subfolder: auth
🔍 DEBUG_CHECKPOINT_A4b: Checking for nested schema layout in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth
🔍 DEBUG_CHECKPOINT_A4c: Found nested steps dir: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps (display: auth/login)
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: auth/login|/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
[INFO] 📚 Detected grouped table folders under identity/: auth/login

🔍 DEBUG_CHECKPOINT_A7: Current IDENTIFIER=coordinator
🔍 DEBUG_CHECKPOINT_A8_PROCEED: Processing tables on coordinator/main node
🔍 DEBUG_CHECKPOINT_A9: Processing table: auth/login at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
[INFO] 🔸 Table group: auth/login
🔍 DEBUG_CHECKPOINT_A10: About to run numbered steps for table: auth/login
🔍 DEBUG_CHECKPOINT_B1: run_all_numbered_steps_in_dir called for dir=/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps table=auth/login
🔍 DEBUG_CHECKPOINT_B2: Found 1 numbered steps: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B3: About to run step: 01-init-schema.sh
Ab substep 0 compelete start
[DEBUG] Tracking substep start: steps/01-install/steps/identity/auth/login/01-init-schema (RUN_UUID=9145d5f5-4202-4699-a36c-b610c6461ede)
Ab substep 0 compelete start
[INFO] 📦 01 init schema...
Ab substep 1 compelete start
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing auth.login_account table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating auth schema...
NOTICE:  schema "auth" already exists, skipping
CREATE SCHEMA
✅ Schema created
🔧 Creating account_status ENUM...
DO
✅ ENUM created
🔧 Creating auth.login_account table...
NOTICE:  relation "login_account" already exists, skipping
CREATE TABLE
✅ Table created (Citus-compatible with region_hint in all constraints)
🔧 Creating indexes...
NOTICE:  relation "idx_login_account_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_login_account_username" already exists, skipping
CREATE INDEX
✅ Indexes created
ℹ️  Table already registered with Citus
🎉 Schema initialization complete for fastorder_identity_sau_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

📊 Registering environment in monitoring database (obs schema)...
   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
   Resource IP: 142.93.238.16
⚠️  Could not connect to monitoring database, skipping registration
   You can manually register later using:
   /opt/fastorder/bash/scripts/env_app_setup/setup/04-postgresql/steps/register-authN-af-aaaa1-dev.sh

==========================================
✅ Schema initialization complete!
==========================================
Ab substep 1 compelete end
Ab substep 2 compelete start
Ab substep 2 compelete end

🔍 DEBUG_CHECKPOINT_B4: Completed step: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B5: All numbered steps completed for /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A11: Completed numbered steps for table: auth/login
compeleted here

🔍 DEBUG_CHECKPOINT_A12: All tables processed
End of 04-postgresql/steps/01-install/steps/identity/run.sh

[0;32m✓[0m ✅ Standby worker-01-standby-01 setup completed

[0;32m✓[0m ✅ PostgreSQL installation completed
[0;34m[INFO][0m Discovering additional setup steps...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 02-pg-bouncer.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up PgBouncer connection pooling...
[2026-01-02 11:03:54 UTC] USER=www-data EUID=0 PID=1975822 ACTION=fsop ARGS=rm -f /tmp/pgbouncer-ip.service /tmp/pgbouncer.service
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[0;32m✓ [SECRETS][0m Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[0;34m[SECRETS][0m Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[0;34m[SECRETS][0m            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[0;34m[SECRETS][0m            Backups (build_backup_path)
[0;34m[SECRETS][0m Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[0;34m[INFO][0m Checking for existing PgBouncer application environment in topology …
[0;32m[OK][0m   Using existing PgBouncer environment:
[0;34m[INFO][0m   IP:     10.100.1.204
[0;34m[INFO][0m   FQDN:   db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[0;34m[INFO][0m   Domain: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[0;34m[INFO][0m Ensuring /etc/hosts entry for db-identity-sau-main-dev-postgresql-bouncer.fastorder.com …
[0;32m[OK][0m   /etc/hosts already contains entry for db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[1;33m[WARN][0m IP 10.100.1.204 is assigned to multiple interfaces:
    inet 10.100.1.103/32 scope global lo
       valid_lft forever preferred_lft forever
    inet 10.100.1.204/32 scope global lo:pgbouncer
--
    inet 10.100.1.214/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.100.1.204/32 scope global eth0:pgbouncer
[1;33m[WARN][0m This may cause routing issues
[0;34m[INFO][0m Final verification of /etc/hosts entry for db-identity-sau-main-dev-postgresql-bouncer.fastorder.com …
[0;32m[OK][0m   /etc/hosts correctly maps db-identity-sau-main-dev-postgresql-bouncer.fastorder.com to 10.100.1.204
[0;32m[OK][0m   PgBouncer IP 10.100.1.204 already correctly bound to lo:pgbouncer
[2026-01-02 11:03:55 UTC] USER=www-data EUID=0 PID=1975903 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 11:03:57 UTC] USER=www-data EUID=0 PID=1976006 ACTION=passthru ARGS=systemctl restart pgbouncer-ip@identity-sau-main-dev.service
[2026-01-02 11:03:57 UTC] USER=www-data EUID=0 PID=1976017 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer-ip@identity-sau-main-dev.service
[0;32m[OK][0m   pgbouncer-ip@identity-sau-main-dev.service is active
[2026-01-02 11:03:57 UTC] USER=www-data EUID=0 PID=1976041 ACTION=fsop ARGS=mkdir -p /etc/pgbouncer/identity-sau-main-dev
[2026-01-02 11:03:57 UTC] USER=www-data EUID=0 PID=1976050 ACTION=fsop ARGS=mkdir -p /run/pgbouncer/identity-sau-main-dev
[2026-01-02 11:03:57 UTC] USER=www-data EUID=0 PID=1976059 ACTION=fsop ARGS=mkdir -p /var/log/pgbouncer/identity-sau-main-dev
[2026-01-02 11:03:57 UTC] USER=www-data EUID=0 PID=1976068 ACTION=fsop ARGS=chmod 750 /etc/pgbouncer/identity-sau-main-dev
[2026-01-02 11:03:57 UTC] USER=www-data EUID=0 PID=1976077 ACTION=fsop ARGS=chmod 750 /run/pgbouncer/identity-sau-main-dev
[2026-01-02 11:03:57 UTC] USER=www-data EUID=0 PID=1976086 ACTION=fsop ARGS=chmod 750 /var/log/pgbouncer/identity-sau-main-dev
[2026-01-02 11:03:57 UTC] USER=www-data EUID=0 PID=1976095 ACTION=fsop ARGS=chown root:postgres /etc/pgbouncer/identity-sau-main-dev
[2026-01-02 11:03:57 UTC] USER=www-data EUID=0 PID=1976104 ACTION=fsop ARGS=chown postgres:postgres /run/pgbouncer/identity-sau-main-dev
[2026-01-02 11:03:57 UTC] USER=www-data EUID=0 PID=1976113 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbouncer/identity-sau-main-dev
[0;34m[INFO][0m Generating pgbouncer_admin client certificates...
[0;34m[INFO][0m ⏳ This may take 30-60 seconds...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    pgbouncer_admin
Identifier:  pgbouncer
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        pgbouncer
  User (CN):   pgbouncer_admin
  Hostname:    db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:03:58 UTC] USER=www-data EUID=0 PID=1976150 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-pgbouncer-pgbouncer_admin
[2026-01-02 11:03:58 UTC] USER=www-data EUID=0 PID=1976159 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-02 11:03:58 UTC] USER=www-data EUID=0 PID=1976168 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
[2026-01-02 11:03:58 UTC] USER=www-data EUID=0 PID=1976179 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-02 11:03:58 UTC] USER=www-data EUID=0 PID=1976188 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = pgbouncer_admin
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-02 11:03:58 UTC] USER=www-data EUID=0 PID=1976203 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-02 11:03:58 UTC] USER=www-data EUID=0 PID=1976212 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-02 11:03:58 UTC] USER=www-data EUID=0 PID=1976221 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 11:03:58 UTC] USER=www-data EUID=0 PID=1976230 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 11:03:58 UTC] USER=www-data EUID=0 PID=1976248 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 11:03:58 UTC] USER=www-data EUID=0 PID=1976257 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976266 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976275 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976284 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976293 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976302 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976311 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976320 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976329 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976339 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976348 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976357 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976389 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976398 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976407 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976416 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976425 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976434 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976443 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976452 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976461 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976471 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976481 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976491 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:03:59 UTC] USER=www-data EUID=0 PID=1976501 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976510 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976519 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976528 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976537 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976546 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976555 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976565 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976577 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976586 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976595 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976605 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976615 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976631 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976640 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976649 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976658 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976667 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976676 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976685 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976695 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976704 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 11:04:00 UTC] USER=www-data EUID=0 PID=1976713 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976723 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976737 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976746 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976755 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976764 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976773 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976782 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976792 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976801 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976811 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976820 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976829 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976839 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976849 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976868 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976883 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976895 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976904 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976916 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976925 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:04:01 UTC] USER=www-data EUID=0 PID=1976934 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-02 11:04:02 UTC] USER=www-data EUID=0 PID=1976944 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: pgbouncer_admin
Node: pgbouncer
FQDN: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-bouncer.fastorder.com -U pgbouncer_admin -d postgres

[0;32m[OK][0m   mTLS client certificate present: /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[0;34m[INFO][0m Creating symlinks to canonical certificates in /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend...
[2026-01-02 11:04:02 UTC] USER=www-data EUID=0 PID=1976961 ACTION=fsop ARGS=mkdir -p /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend
[2026-01-02 11:04:02 UTC] USER=www-data EUID=0 PID=1976970 ACTION=fsop ARGS=mkdir -p /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend
[2026-01-02 11:04:02 UTC] USER=www-data EUID=0 PID=1976979 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-01-02 11:04:02 UTC] USER=www-data EUID=0 PID=1976988 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[2026-01-02 11:04:02 UTC] USER=www-data EUID=0 PID=1976997 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt
[0;34m[INFO][0m Creating coordinator CA symlink for PostgreSQL server verification...
[2026-01-02 11:04:02 UTC] USER=www-data EUID=0 PID=1977008 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m Verifying canonical certificate permissions...
[2026-01-02 11:04:02 UTC] USER=www-data EUID=0 PID=1977018 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 11:04:02 UTC] USER=www-data EUID=0 PID=1977027 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 11:04:02 UTC] USER=www-data EUID=0 PID=1977036 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-02 11:04:02 UTC] USER=www-data EUID=0 PID=1977045 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[0;32m[OK][0m   Backend certificate symlinks created in /etc/ssl
[0;32m[OK][0m   Coordinator CA symlink created for server verification
[0;32m[OK][0m   Certificates already in canonical location - no symlinks needed
[2026-01-02 11:04:02 UTC] USER=www-data EUID=0 PID=1977056 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.crt
[2026-01-02 11:04:02 UTC] USER=www-data EUID=0 PID=1977065 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.key
[2026-01-02 11:04:02 UTC] USER=www-data EUID=0 PID=1977074 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-02 11:04:02 UTC] USER=www-data EUID=0 PID=1977083 ACTION=fsop ARGS=test -r /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m PgBouncer will use PostgreSQL coordinator CA: /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;32m[OK][0m   PostgreSQL coordinator at db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432 is reachable
[0;34m[INFO][0m Dumping SCRAM secrets from coordinator for PgBouncer auth_file …
[2026-01-02 11:04:03 UTC] USER=www-data EUID=0 PID=1977104 ACTION=fsop ARGS=cp /tmp/tmp.M8ilxgutgI /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-02 11:04:03 UTC] USER=www-data EUID=0 PID=1977113 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-02 11:04:03 UTC] USER=www-data EUID=0 PID=1977122 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[0;32m[OK][0m   Auth file written: /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[0;34m[INFO][0m Retrieved password from vault for pgbouncer_admin
[0;34m[INFO][0m Ensuring PgBouncer admin role 'pgbouncer_admin' exists in Postgres (coordinator) …
[0;32m[OK][0m   Role pgbouncer_admin created/updated successfully
[0;34m[SECRETS][0m Setting credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/pgbouncer_admin
[0;32m✓ [SECRETS][0m Credentials updated in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/pgbouncer_admin
[0;34m[INFO][0m ✅ PgBouncer admin password stored in centralized secrets vault
[0;34m[INFO][0m Re-fetching SCRAM secrets after role creation to ensure pgbouncer_admin is included …
[2026-01-02 11:04:13 UTC] USER=www-data EUID=0 PID=1977227 ACTION=fsop ARGS=cp /tmp/tmp.bVGkN2SKCl /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-02 11:04:13 UTC] USER=www-data EUID=0 PID=1977236 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-02 11:04:13 UTC] USER=www-data EUID=0 PID=1977245 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[0;32m[OK][0m   Auth file updated with pgbouncer_admin SCRAM hash
[0;34m[INFO][0m Auth file contains [2026-01-02 11:04:13 UTC] USER=www-data EUID=0 PID=1977255 ACTION=passthru ARGS=bash -c wc -l < '/etc/pgbouncer/identity-sau-main-dev/userlist.txt'
4 user(s)
[0;32m[OK][0m   Admin 'pgbouncer_admin' password generated and saved
[0;34m[INFO][0m Configuring PostgreSQL to prevent Citus metadata sync hangs...
ALTER ROLE
[0;32m[OK][0m   Disabled Citus metadata sync for pgbouncer_admin
[0;34m[INFO][0m Verifying application database fastorder_identity_sau_main_dev_db exists...
[0;32m[OK][0m   ✓ Database fastorder_identity_sau_main_dev_db exists
[0;34m[INFO][0m Granting permissions to pgbouncer_admin on fastorder_identity_sau_main_dev_db...
GRANT
[0;32m[OK][0m   ✓ Granted CONNECT on fastorder_identity_sau_main_dev_db to pgbouncer_admin
GRANT
[0;32m[OK][0m   ✓ Granted USAGE on schema public to pgbouncer_admin
GRANT
[0;32m[OK][0m   ✓ Granted SELECT on all tables to pgbouncer_admin
ALTER DATABASE
[0;32m[OK][0m   Set synchronous_commit=local for fastorder_identity_sau_main_dev_db
[0;34m[INFO][0m Ensuring pg_hba.conf entry for pgbouncer_admin …
[0;34m[INFO][0m Adding pg_hba.conf entries for pgbouncer_admin with cert auth …
[2026-01-02 11:04:14 UTC] USER=unknown EUID=33 PID=1977290 ACTION=-u ARGS=postgres bash
ERROR: Invalid or unauthorized action: -u
[0;32m[OK][0m   pg_hba.conf updated and PostgreSQL configuration reloaded
[1;33m[WARN][0m pg_hba.conf entry may not have loaded correctly
[0;34m[INFO][0m Writing /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini …
[2026-01-02 11:04:15 UTC] USER=www-data EUID=0 PID=1977319 ACTION=fsop ARGS=cp /tmp/tmp.BJgMrRbSLM /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
[2026-01-02 11:04:15 UTC] USER=www-data EUID=0 PID=1977328 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
[2026-01-02 11:04:16 UTC] USER=www-data EUID=0 PID=1977337 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
[2026-01-02 11:04:16 UTC] USER=www-data EUID=0 PID=1977346 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbouncer/identity-sau-main-dev /run/pgbouncer/identity-sau-main-dev /var/log/pgbouncer/identity-sau-main-dev
[2026-01-02 11:04:16 UTC] USER=www-data EUID=0 PID=1977355 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[0;32m[OK][0m   pgbouncer.ini ready
[0;34m[INFO][0m Verifying TLS settings in pgbouncer.ini:
[2026-01-02 11:04:16 UTC] USER=www-data EUID=0 PID=1977365 ACTION=fsop ARGS=grep -E (client_tls_sslmode|server_tls) /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
client_tls_sslmode = verify-full
server_tls_sslmode = verify-full
server_tls_ca_file = /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
server_tls_cert_file = /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
server_tls_key_file  = /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;34m[INFO][0m Verifying PgBouncer server certificate files:
[2026-01-02 11:04:16 UTC] USER=www-data EUID=0 PID=1977375 ACTION=fsop ARGS=test -r /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[0;32m[OK][0m   Server cert readable by postgres: /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-01-02 11:04:16 UTC] USER=www-data EUID=0 PID=1977385 ACTION=fsop ARGS=test -r /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;32m[OK][0m   Server key readable by postgres: /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;34m[INFO][0m Verifying coordinator CA certificate:
[2026-01-02 11:04:16 UTC] USER=www-data EUID=0 PID=1977394 ACTION=fsop ARGS=test -r /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;32m[OK][0m   Coordinator CA readable by postgres: /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m Preflight: stopping any conflicting PgBouncer on 6432 …
[2026-01-02 11:04:16 UTC] USER=www-data EUID=0 PID=1977403 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer.service
[2026-01-02 11:04:16 UTC] USER=www-data EUID=0 PID=1977412 ACTION=passthru ARGS=systemctl stop pgbouncer@identity-sau-main-dev.service
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json?all=1": dial unix /var/run/docker.sock: connect: permission denied
[1;33m[WARN][0m Killing existing pgbouncer processes: 1813344
1813393
[2026-01-02 11:04:16 UTC] USER=www-data EUID=0 PID=1977438 ACTION=passthru ARGS=bash -c kill -9 1813344
[2026-01-02 11:04:16 UTC] USER=www-data EUID=0 PID=1977450 ACTION=passthru ARGS=bash -c kill -9 1813393
[2026-01-02 11:04:19 UTC] USER=www-data EUID=0 PID=1977486 ACTION=passthru ARGS=systemctl daemon-reload
[0;32m[OK][0m   systemd unit installed: pgbouncer@identity-sau-main-dev.service
[0;34m[INFO][0m Running pre-flight IP conflict check for 10.100.1.204:6432 …
[1;33m[WARN][0m IP conflict checker not found at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/check-ip-conflicts.sh
[1;33m[WARN][0m Skipping pre-flight check - conflicts may occur
[0;34m[INFO][0m Starting PgBouncer (identity-sau-main-dev) …
[2026-01-02 11:04:20 UTC] USER=www-data EUID=0 PID=1977588 ACTION=passthru ARGS=systemctl restart pgbouncer@identity-sau-main-dev.service
[2026-01-02 11:04:20 UTC] USER=www-data EUID=0 PID=1977599 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer@identity-sau-main-dev.service
[0;32m[OK][0m   Service ACTIVE
[0;34m[INFO][0m Verifying auth_file before probing …
[0;34m[INFO][0m Auth file contains 4 user(s)
[1;33m[WARN][0m Auth file does NOT contain pgbouncer_admin entry - authentication will fail
[0;34m[INFO][0m Probing admin console via SSL (psql to database 'pgbouncer') …
[0;34m[INFO][0m Retrieved password from vault for admin console probe
[1;33m[WARN][0m SSL connection issue detected
[0;34m[INFO][0m Attempting connection with sslmode=disable for testing...
[1;33m[WARN][0m If this fails, check PgBouncer client_tls_sslmode setting
[1;33m[WARN][0m Admin console probe failed (see error below)
psql: error: connection to server at "10.100.1.204", port 6432 failed: SSL error: certificate verify failed
[1;33m[WARN][0m Troubleshooting:
[1;33m[WARN][0m   1. Check auth_file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[1;33m[WARN][0m   2. Test with: PGPASSWORD='kppzNMG6WDrJWGUYcBARr4ME' psql -h 10.100.1.204 -p 6432 -U pgbouncer_admin -d pgbouncer
[1;33m[WARN][0m   3. Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@identity-sau-main-dev.service -n 50

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m   Running Comprehensive PgBouncer Verification Tests
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[0;34m[INFO][0m Password extracted: kppzNMG6WD... (using postgres user certificates)

[0;34m[INFO][0m Test 1/7: Admin Console - SHOW POOLS
[1;33m[WARN][0m ✗ SHOW POOLS: FAILED
[1;33m[WARN][0m Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@identity-sau-main-dev.service -n 50

[0;34m[INFO][0m Test 2/7: Admin Console - SHOW VERSION
[1;33m[WARN][0m ✗ SHOW VERSION: FAILED

[0;34m[INFO][0m Test 3/7: Admin Console - SHOW STATS
[1;33m[WARN][0m ✗ SHOW STATS: FAILED

[0;34m[INFO][0m Test 4/7: Admin Console - SHOW DATABASES
[1;33m[WARN][0m ✗ SHOW DATABASES: FAILED

[0;34m[INFO][0m Test 5/7: Admin Console - SHOW CONFIG
[1;33m[WARN][0m ✗ SHOW CONFIG: FAILED
psql   "host=db-identity-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_identity_sau_main_dev_db user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME    connect_timeout=5 sslmode=verify-full    sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt    sslcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt    sslkey=/etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key"   --no-psqlrc -Atc 'SELECT version();'

[0;34m[INFO][0m Test 6/7: Application Database - SELECT version()
[1;33m[WARN][0m ✗ Application database query: FAILED (timeout or connection issue)
[1;33m[WARN][0m    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[0;34m[INFO][0m Test 7/8: Application Database - Connection Details
[1;33m[WARN][0m ✗ Connection details: FAILED (timeout or connection issue)
[1;33m[WARN][0m    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[0;34m[INFO][0m Test 8/8: End-to-End Application Routing - Pool Verification
[0;34m[INFO][0m   Running actual queries through PgBouncer to verify routing and pooling...
[1;33m[WARN][0m ✗ End-to-end routing verification: FAILED - All 3 queries failed
[1;33m[WARN][0m    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[1;33m[WARN][0m    Otherwise check if database fastorder_identity_sau_main_dev_db exists and user pgbouncer_admin has permissions

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m   Verification Complete - Tests 1-5 PASSED (Admin console verified)
[1;33m[WARN][0m   Tests 6-8 FAILED - Application database not accessible
[1;33m[WARN][0m   This is expected if Citus is not set up yet
[1;33m[WARN][0m   Run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[0;32m[OK][0m   PgBouncer is up for identity-sau-main-dev

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Connection Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Password stored in: AWS Secrets Manager (fastorder/db/web/ksa/main/dev/postgresqlidentity/sau/main/dev/coordinator-pgbouncer_admin)
Current password: kppzNMG6WDrJWGUYcBARr4ME

1. Admin Console (using IP address to avoid DNS/SSL issues):
   psql "host=10.100.1.204 port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME sslmode=verify-full sslrootcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

2. Admin Console (using hostname):
   psql "host=db-identity-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME sslmode=verify-full sslrootcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

3. Application Database:
   psql "host=db-identity-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_identity_sau_main_dev_db sslkey=/etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME sslmode=verify-full sslrootcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

4. Using .pgpass file:
   echo "db-identity-sau-main-dev-postgresql-bouncer.fastorder.com:6432:*:pgbouncer_admin:kppzNMG6WDrJWGUYcBARr4ME" >> ~/.pgpass
   chmod 600 ~/.pgpass
   psql -h db-identity-sau-main-dev-postgresql-bouncer.fastorder.com -p 6432 -U pgbouncer_admin -d fastorder_identity_sau_main_dev_db

5. Retrieve password from vault:
   source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
   PGPASSWORD="$(get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password')" \
     psql -h 10.100.1.204 -p 6432 -U pgbouncer_admin -d pgbouncer -c "SHOW POOLS;"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Architecture
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  • Default db 'fastorder_identity_sau_main_dev_db' → Citus coordinator (db-identity-sau-main-dev-postgresql-coordinator.fastorder.com)
  • Worker access: 'fastorder_identity_sau_main_dev_db_worker_1', 'fastorder_identity_sau_main_dev_db_worker_2', … (if exist)
  • Client TLS: require (password auth) / verify-full (mTLS with certs)
  • Server TLS: verify-full (PgBouncer validates PostgreSQL certs)
  • Auth: SCRAM-SHA-256 via /etc/pgbouncer/identity-sau-main-dev/userlist.txt
  • Pool mode: transaction (stateless connections)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Management
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Service Status:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer@identity-sau-main-dev.service
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@identity-sau-main-dev.service

Logs:
  command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@identity-sau-main-dev.service -f
  /usr/local/bin/fastorder-provisioning-wrapper.sh tail -f /var/log/pgbouncer/identity-sau-main-dev/pgbouncer.log

Reload Config:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@identity-sau-main-dev.service

Restart:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@identity-sau-main-dev.service

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Files
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Config:        /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
Auth file:     /etc/pgbouncer/identity-sau-main-dev/userlist.txt
Server cert:   /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.crt
Server key:    /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.key
CA cert:       /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt
PG CA:         /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
Logs:          /var/log/pgbouncer/identity-sau-main-dev/pgbouncer.log

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Troubleshooting
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


If "SASL authentication failed":
  1. Check auth file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/identity-sau-main-dev/userlist.txt
  2. Verify pgbouncer_admin is present with SCRAM hash
  3. Get password from vault:
     source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
     get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password'
  4. Reload PgBouncer: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@identity-sau-main-dev.service

If "no pg_hba.conf entry":
  1. Check pg_hba.conf on coordinator
  2. Add rule: hostssl all pgbouncer_admin 10.100.1.204/32 cert clientcert=verify-full
  3. Reload PostgreSQL

To add users to PgBouncer:
  1. Create user in PostgreSQL with password
  2. Re-run SCRAM dump:
     psql "host=db-identity-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 dbname=postgres user=postgres \
       sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt \
       sslcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt sslkey=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key" \
       -Atc "SELECT '\"' || rolname || '\" \"' || rolpassword || '\"' \
             FROM pg_authid WHERE rolpassword LIKE 'SCRAM-SHA-256%' \
             AND rolcanlogin ORDER BY rolname;" | command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop tee /etc/pgbouncer/identity-sau-main-dev/userlist.txt
  3. Reload: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@identity-sau-main-dev.service

[0;34m[INFO][0m Registering PgBouncer node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        identity-sau-main-dev-pgbouncer
[INFO]   Identifier Parent: postgresql
[INFO]   IP:                10.100.1.204
[INFO]   Port:              6432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 426480a5-2f64-4fc0-b2b5-710f9ccb059a
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[0;32m[OK][0m   PgBouncer node registered to observability API
[0;32m✓[0m ✅ PgBouncer setup completed

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 03-citus-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m CITUS DISTRIBUTED CLUSTER SETUP
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Phase 1: Installing Citus extension on workers...
[0;34m[INFO][0m Phase 2: Setting up coordinator and registering workers...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[0;34m[INFO][0m 📦 PHASE 1: Installing Citus extension on 1 worker(s)...

[0;34m[INFO][0m → Worker 1/1: Installing Citus on worker-01...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS CLUSTER SETUP
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════

[0;34m[INFO][0m 🔧 Setting up Citus Worker...
[0;34m[INFO][0m Temporarily disabling synchronous replication for extension installation...
t
[0;34m[INFO][0m Installing Citus extension on worker...
[0;32m[OK][0m   Citus extension installed on worker
[0;34m[INFO][0m Restoring synchronous replication settings...
t
[0;34m[INFO][0m Worker Citus extension installed - registration will happen when coordinator setup runs

[0;32m[OK][0m   Citus setup complete for worker-01
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m✓[0m   ✅ Citus extension installed on worker-01

[0;32m✓[0m ✅ Phase 1 Complete: All 1 workers have Citus extension installed

[0;34m[INFO][0m 🔧 PHASE 2: Setting up Citus coordinator and registering workers...

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS CLUSTER SETUP
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════

[0;34m[INFO][0m 🔧 Setting up Citus Coordinator...

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m DIAGNOSTIC: Configuration Variables
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m PG_WORKERS_NUM: 1
[0;34m[INFO][0m ENV_ID: identity-sau-main-dev
[0;34m[INFO][0m DOMAIN: fastorder.com
[0;34m[INFO][0m PORT: 5432
[0;34m[INFO][0m SOCKET_DIR: /var/run/postgresql-identity-sau-main-dev-coordinator
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[0;34m[INFO][0m Ensuring postgres client certificates exist for coordinator...
[0;32m[OK][0m   Postgres client certificates already exist for coordinator
[0;34m[INFO][0m Adding citus_cert_map to coordinator pg_ident.conf...
[0;32m[OK][0m   pg_ident.conf updated for coordinator
[0;34m[INFO][0m Installing Citus extension on coordinator...
[0;32m[OK][0m   Citus extension installed on coordinator (postgres database)
[0;34m[INFO][0m Installing Citus extension on application database: fastorder_identity_sau_main_dev_db...
[0;32m[OK][0m   Citus extension installed on application database: fastorder_identity_sau_main_dev_db
[0;34m[INFO][0m Configuring Citus SSL connection parameters...
[2026-01-02 11:04:40 UTC] USER=www-data EUID=0 PID=1978293 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
[0;32m[OK][0m   ✅ Citus SSL connection parameters configured: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[1;33m[WARN][0m Node not identified as coordinator, initializing...
[0;34m[INFO][0m Checking coordinator configuration...
[0;34m[INFO][0m Persisting citus.local_hostname to postgresql.conf...
[2026-01-02 11:04:43 UTC] USER=www-data EUID=0 PID=1978337 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /var/lib/postgresql/17/identity-sau-main-dev/coordinator/postgresql.conf
[2026-01-02 11:04:43 UTC] USER=www-data EUID=0 PID=1978358 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
[0;32m[OK][0m   ✅ citus.local_hostname persisted to config and reloaded
[0;34m[INFO][0m Configuring coordinator hostname in postgres database: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432

[0;32m[OK][0m   ✅ Coordinator hostname set to db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432 in postgres database
[0;34m[INFO][0m Checking coordinator configuration in application database: fastorder_identity_sau_main_dev_db...
[1;33m[WARN][0m ⚠️  Coordinator registered as 'localhost' in application database, fixing...
[0;34m[INFO][0m Configuring coordinator hostname in application database: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432
[0;32m[OK][0m   ✅ Coordinator hostname set to db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432 in application database
[0;34m[INFO][0m Validating coordinator configuration before worker registration...
[0;32m[OK][0m   ✅ Coordinator hostname validated: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[0;32m[OK][0m   ✅ citus_tables view is accessible
[0;34m[INFO][0m Checking coordinator self-registration...
[0;32m[OK][0m   ✅ Coordinator is already self-registered
[0;34m[INFO][0m Configuring coordinator shard placement policy...
[0;32m[OK][0m   ✅ Coordinator already configured in postgres database (shouldhaveshards = false)
[1;33m[WARN][0m ⚠️  Coordinator has 17 shards in fastorder_identity_sau_main_dev_db - cannot set shouldhaveshards=false
[1;33m[WARN][0m    You must rebalance shards to workers first, then run this setup again
[1;33m[WARN][0m    Skipping shouldhaveshards configuration for application database
[0;34m[INFO][0m Registering 1 worker(s) to Citus cluster...

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m PRE-FLIGHT: Checking worker availability...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Checking worker worker-01...
[0;34m[INFO][0m   FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[0;32m[OK][0m   ✅ Worker worker-01 is reachable via SSL
[0;32m[OK][0m   All workers are reachable - proceeding with registration

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Adding Citus worker: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Adding citus_cert_map to worker-01 pg_ident.conf...
[0;32m[OK][0m   pg_ident.conf updated for worker-01
[0;34m[INFO][0m Configuring worker worker-01 HBA for coordinator (10.100.1.213) access...
[0;32m[OK][0m   Worker worker-01 HBA configured for coordinator (10.100.1.213)
[0;34m[INFO][0m Adding replication rules for 3 standby(s)...
[0;32m[OK][0m   Replication rules already exist for worker-01
[0;34m[INFO][0m Reloading worker worker-01 to apply HBA changes...
[2026-01-02 11:04:46 UTC] USER=www-data EUID=0 PID=1978497 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
[0;34m[INFO][0m Configuring coordinator HBA for worker worker-01 (10.100.1.214) access...
[0;32m[OK][0m   Coordinator HBA configured for worker worker-01 (10.100.1.214)
[0;34m[INFO][0m Reloading coordinator to apply HBA changes...
[2026-01-02 11:04:46 UTC] USER=www-data EUID=0 PID=1978527 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
[0;34m[INFO][0m Ensuring postgres client certificates exist for worker-01...
[0;32m[OK][0m   Postgres client certificates already exist for worker-01
[0;34m[INFO][0m Configuring citus.node_conninfo on worker-01...
[2026-01-02 11:04:47 UTC] USER=www-data EUID=0 PID=1978545 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
[0;32m[OK][0m   citus.node_conninfo configured on worker-01
[0;34m[INFO][0m Temporarily relaxing sync-rep on worker worker-01...
t
[0;32m[OK][0m   Worker worker-01 sync-rep relaxed (was: sync_commit=on)
[0;34m[INFO][0m Ensuring Citus extension on worker databases...
CREATE EXTENSION
CREATE EXTENSION
[0;34m[INFO][0m Running citus_add_node with 180s timeout...
NOTICE:  shards are still on the coordinator after adding the new node
HINT:  Use SELECT rebalance_table_shards(); to balance shards data between workers and coordinator or SELECT citus_drain_node('db-identity-sau-main-dev-postgresql-coordinator.fastorder.com',5432); to permanently move shards away from the coordinator.
2
[0;34m[INFO][0m Restoring worker worker-01 sync-rep settings...
t
[0;32m[OK][0m   Worker worker-01 sync-rep restored
[0;32m[OK][0m   ✅ Worker db-identity-sau-main-dev-postgresql-worker-01.fastorder.com successfully added to Citus cluster
[0;34m[INFO][0m    Node ID: 2
[0;34m[INFO][0m    Registered in: postgres, fastorder_identity_sau_main_dev_db
[0;32m[OK][0m   Worker worker-01 registration successful
[0;34m[INFO][0m Configuring worker worker-01 shard placement policy...
[0;32m[OK][0m   ✅ Worker worker-01 configured to hold shards in all databases


[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m POST-REGISTRATION: Verifying cluster state...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Expected workers: 1
[0;34m[INFO][0m Registered workers: 1
[0;32m[OK][0m   ✅ All 1 workers successfully registered!

[0;34m[INFO][0m Citus cluster configuration:
db-identity-sau-main-dev-postgresql-coordinator.fastorder.com  5432  0  t  primary  f
db-identity-sau-main-dev-postgresql-worker-01.fastorder.com    5432  1  t  primary  t

[0;34m[INFO][0m Note: groupid=0 is the coordinator, groupid>0 are workers
[0;34m[INFO][0m       shouldhaveshards: false=query router only, true=holds data shards

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m FINAL VALIDATION: Verifying configuration persistence...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:04:51 UTC] USER=www-data EUID=0 PID=1978714 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /var/lib/postgresql/17/identity-sau-main-dev/coordinator/postgresql.conf
[0;32m[OK][0m   ✅ citus.local_hostname persisted in postgresql.conf
[0;32m[OK][0m   ✅ All 1 worker(s) successfully registered and verified

[0;32m[OK][0m   ✅ All validation checks passed
[0;32m[OK][0m   Citus coordinator setup complete

[0;32m[OK][0m   Citus setup complete for coordinator
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════

[0;32m✓[0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✓[0m ✅ CITUS CLUSTER SETUP COMPLETED SUCCESSFULLY
[0;32m✓[0m    Coordinator: Ready and accepting connections
[0;32m✓[0m    Workers registered: 1
[0;32m✓[0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 05-backup-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up coordinator backup...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for identity-sau-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-01-02 11:04:53 UTC] USER=www-data EUID=0 PID=1978780 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-02 11:04:53 UTC] USER=www-data EUID=0 PID=1978789 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/identity-sau-main-dev
[2026-01-02 11:04:53 UTC] USER=www-data EUID=0 PID=1978798 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-01-02 11:04:53 UTC] USER=www-data EUID=0 PID=1978807 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-02 11:04:53 UTC] USER=www-data EUID=0 PID=1978816 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-02 11:04:53 UTC] USER=www-data EUID=0 PID=1978825 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-02 11:05:02 UTC] USER=www-data EUID=0 PID=1978915 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-02 11:05:02 UTC] USER=www-data EUID=0 PID=1978924 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-02 11:05:02 UTC] USER=www-data EUID=0 PID=1978935 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-01-02 11:05:02 UTC] USER=www-data EUID=0 PID=1978944 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-02 11:05:02 UTC] USER=www-data EUID=0 PID=1978953 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-identity-sau-main-dev
[2026-01-02 11:05:03 UTC] USER=www-data EUID=0 PID=1978975 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-02 11:05:03 UTC] USER=www-data EUID=0 PID=1978985 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-01-02 11:05:03 UTC] USER=www-data EUID=0 PID=1978996 ACTION=fsop ARGS=find /var/lib/postgresql/17/identity-sau-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-01-02 11:05:03 UTC] USER=www-data EUID=0 PID=1979006 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-02 11:05:03 UTC] USER=www-data EUID=0 PID=1979015 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-01-02 11:05:03 UTC] USER=www-data EUID=0 PID=1979024 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-01-02 11:05:03 UTC] USER=www-data EUID=0 PID=1979033 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-02 11:05:03 UTC] USER=www-data EUID=0 PID=1979047 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/PG_VERSION
[2026-01-02 11:05:03 UTC] USER=www-data EUID=0 PID=1979058 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza identity-sau-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-02 11:05:04 UTC] USER=www-data EUID=0 PID=1979122 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-02 11:05:07 UTC] USER=www-data EUID=0 PID=1979145 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 11:05:11 UTC] USER=www-data EUID=0 PID=1979193 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-02 11:05:11 UTC] USER=www-data EUID=0 PID=1979218 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator --log-level-console=info check
2026-01-02 11:05:11.881 P00   INFO: check command begin 2.56.0: --exec-id=1979226-08ca2136 --log-level-console=info --log-level-file=debug --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-02 11:05:11.947 P00   INFO: check repo1 configuration (primary)
2026-01-02 11:05:11.975 P00  ERROR: [028]: backup and archive info files exist but do not match the database
                                    HINT: is this the correct stanza?
                                    HINT: did an error occur during stanza-upgrade?
2026-01-02 11:05:11.976 P00   INFO: check command end: aborted with exception [028]
[WARN] ⚠️  Stanza verification failed - this may be normal if WAL archiving hasn't started yet
[WARN]    The backup system is configured and will work once WAL segments are generated

[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-02 11:05:12 UTC] USER=www-data EUID=0 PID=1979244 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-02 11:05:12 UTC] USER=www-data EUID=0 PID=1979255 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-02 11:05:12 UTC] USER=www-data EUID=0 PID=1979273 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[2026-01-02 11:05:12 UTC] USER=www-data EUID=0 PID=1979282 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-02 11:05:12 UTC] USER=www-data EUID=0 PID=1979300 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-identity-sau-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-01-02 11:05:12 UTC] USER=www-data EUID=0 PID=1979320 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 11:05:12 UTC] USER=www-data EUID=0 PID=1979329 ACTION=fsop ARGS=sed -i s|__ENV_ID__|identity-sau-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 11:05:12 UTC] USER=www-data EUID=0 PID=1979338 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/var/lib/postgresql/17/identity-sau-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 11:05:12 UTC] USER=www-data EUID=0 PID=1979347 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 11:05:12 UTC] USER=www-data EUID=0 PID=1979356 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-02 11:05:12.753 P00   INFO: start command begin 2.56.0: --exec-id=1979377-772e7223 --log-level-console=info --log-level-file=debug --stanza=identity-sau-main-dev-coordinator
2026-01-02 11:05:12.754 P00   WARN: stop file does not exist for stanza identity-sau-main-dev-coordinator
2026-01-02 11:05:12.754 P00   INFO: start command end: completed successfully (14ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-02 11:05:12.841 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=1979390-7b4157ec --log-level-console=info --log-level-file=debug --no-online --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-02 11:05:12.845 P00   INFO: stanza-upgrade for stanza 'identity-sau-main-dev-coordinator' on repo1
2026-01-02 11:05:12.868 P00   INFO: stanza-upgrade command end: completed successfully (33ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-02 11:05:12 UTC] USER=www-data EUID=0 PID=1979394 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260102-110512.log
[2026-01-02 11:05:12 UTC] USER=www-data EUID=0 PID=1979403 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260102-110512.log
[2026-01-02 11:05:12 UTC] USER=www-data EUID=0 PID=1979412 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260102-110512.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-01-02 11:05:24 UTC] USER=www-data EUID=0 PID=1979499 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-1978754.log /var/log/pgbackrest/initial-backup-20260102-110512.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260102-110512.log
   2026-01-02 11:05:24.303 P00   INFO: repo1: remove expired backup 20260102-082153F
   2026-01-02 11:05:24.465 P00   INFO: repo1: 17-24 remove archive, start = 000000010000000000000004, stop = 000000010000000000000006
   2026-01-02 11:05:24.466 P00   INFO: repo1: 17-25 no archive to remove
   2026-01-02 11:05:24.467 P00   INFO: repo1: 17-26 remove archive, start = 000000010000000000000003, stop = 000000010000000000000003
   2026-01-02 11:05:24.467 P00   INFO: expire command end: completed successfully (192ms)

[INFO] Current backups:
stanza: identity-sau-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000007/00000001000000000000000B

        full backup: 20260102-082225F
            timestamp start/stop: 2026-01-02 08:22:25+00 / 2026-01-02 08:22:32+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

    db (prior)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000018

        full backup: 20260102-085913F
            timestamp start/stop: 2026-01-02 08:59:13+00 / 2026-01-02 08:59:24+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.3MB, database backup size: 37.3MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

        full backup: 20260102-085944F
            timestamp start/stop: 2026-01-02 08:59:44+00 / 2026-01-02 08:59:49+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.3MB, database backup size: 37.3MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

    db (current)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000004

        full backup: 20260102-110513F
            timestamp start/stop: 2026-01-02 11:05:13+00 / 2026-01-02 11:05:24+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.3MB, database backup size: 37.3MB
            repo1: backup set size: 5.8MB, backup size: 5.8MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         identity-sau-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/identity-sau-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up worker backups for 1 worker(s)...
[0;34m[INFO][0m Setting up backup for: worker-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for identity-sau-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-01-02 11:05:25 UTC] USER=www-data EUID=0 PID=1979556 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-02 11:05:25 UTC] USER=www-data EUID=0 PID=1979567 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/identity-sau-main-dev
[2026-01-02 11:05:25 UTC] USER=www-data EUID=0 PID=1979576 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-01-02 11:05:25 UTC] USER=www-data EUID=0 PID=1979585 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-02 11:05:25 UTC] USER=www-data EUID=0 PID=1979594 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-02 11:05:25 UTC] USER=www-data EUID=0 PID=1979603 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-02 11:05:33 UTC] USER=www-data EUID=0 PID=1979664 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-02 11:05:33 UTC] USER=www-data EUID=0 PID=1979673 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-02 11:05:33 UTC] USER=www-data EUID=0 PID=1979685 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-01-02 11:05:33 UTC] USER=www-data EUID=0 PID=1979702 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-02 11:05:33 UTC] USER=www-data EUID=0 PID=1979713 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-identity-sau-main-dev
[2026-01-02 11:05:34 UTC] USER=www-data EUID=0 PID=1979763 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-02 11:05:34 UTC] USER=www-data EUID=0 PID=1979819 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-01-02 11:05:34 UTC] USER=www-data EUID=0 PID=1979887 ACTION=fsop ARGS=find /var/lib/postgresql/17/identity-sau-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-01-02 11:05:34 UTC] USER=www-data EUID=0 PID=1979971 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-02 11:05:34 UTC] USER=www-data EUID=0 PID=1979980 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-01-02 11:05:34 UTC] USER=www-data EUID=0 PID=1979989 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-02 11:05:34 UTC] USER=www-data EUID=0 PID=1980011 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/PG_VERSION
[2026-01-02 11:05:34 UTC] USER=www-data EUID=0 PID=1980021 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza identity-sau-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-02 11:05:36 UTC] USER=www-data EUID=0 PID=1980089 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-02 11:05:38 UTC] USER=www-data EUID=0 PID=1980213 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[2026-01-02 11:05:42 UTC] USER=www-data EUID=0 PID=1980267 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-02 11:05:42 UTC] USER=www-data EUID=0 PID=1980291 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator --log-level-console=info check
2026-01-02 11:05:43.015 P00   INFO: check command begin 2.56.0: --exec-id=1980298-a32be3bb --log-level-console=info --log-level-file=debug --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-02 11:05:43.053 P00   INFO: check repo1 configuration (primary)
2026-01-02 11:05:43.131 P00   INFO: check repo1 archive for WAL (primary)
2026-01-02 11:05:43.434 P00   INFO: WAL segment 000000010000000000000006 successfully archived to '/var/lib/pgbackrest/backup/identity-sau-main-dev/archive/identity-sau-main-dev-coordinator/17-26/0000000100000000/000000010000000000000006-ba47046163091f04ba85f2a5aea431ae8ff6fdd1.lz4' on repo1
2026-01-02 11:05:43.434 P00   INFO: check command end: completed successfully (456ms)
[INFO] ✅ Stanza verification passed

[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-02 11:05:43 UTC] USER=www-data EUID=0 PID=1980321 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-02 11:05:43 UTC] USER=www-data EUID=0 PID=1980330 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-02 11:05:43 UTC] USER=www-data EUID=0 PID=1980348 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[2026-01-02 11:05:43 UTC] USER=www-data EUID=0 PID=1980357 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-02 11:05:43 UTC] USER=www-data EUID=0 PID=1980375 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-identity-sau-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-01-02 11:05:43 UTC] USER=www-data EUID=0 PID=1980393 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 11:05:43 UTC] USER=www-data EUID=0 PID=1980402 ACTION=fsop ARGS=sed -i s|__ENV_ID__|identity-sau-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 11:05:43 UTC] USER=www-data EUID=0 PID=1980411 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/var/lib/postgresql/17/identity-sau-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 11:05:43 UTC] USER=www-data EUID=0 PID=1980420 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-02 11:05:43 UTC] USER=www-data EUID=0 PID=1980429 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-02 11:05:44.033 P00   INFO: start command begin 2.56.0: --exec-id=1980450-47132108 --log-level-console=info --log-level-file=debug --stanza=identity-sau-main-dev-coordinator
2026-01-02 11:05:44.036 P00   WARN: stop file does not exist for stanza identity-sau-main-dev-coordinator
2026-01-02 11:05:44.036 P00   INFO: start command end: completed successfully (9ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-02 11:05:44.106 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=1980461-b62885d6 --log-level-console=info --log-level-file=debug --no-online --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-02 11:05:44.109 P00   INFO: stanza-upgrade for stanza 'identity-sau-main-dev-coordinator' on repo1
2026-01-02 11:05:44.111 P00   INFO: stanza 'identity-sau-main-dev-coordinator' on repo1 is already up to date
2026-01-02 11:05:44.111 P00   INFO: stanza-upgrade command end: completed successfully (12ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-02 11:05:44 UTC] USER=www-data EUID=0 PID=1980465 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260102-110544.log
[2026-01-02 11:05:44 UTC] USER=www-data EUID=0 PID=1980476 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260102-110544.log
[2026-01-02 11:05:44 UTC] USER=www-data EUID=0 PID=1980485 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260102-110544.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-01-02 11:05:48 UTC] USER=www-data EUID=0 PID=1980536 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-1979524.log /var/log/pgbackrest/initial-backup-20260102-110544.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260102-110544.log
   2026-01-02 11:05:48.369 P00   INFO: repo1: remove expired backup 20260102-082225F
   2026-01-02 11:05:48.450 P00   INFO: repo1: remove archive path /var/lib/pgbackrest/backup/identity-sau-main-dev/archive/identity-sau-main-dev-coordinator/17-24
   2026-01-02 11:05:48.454 P00   INFO: repo1: 17-25 no archive to remove
   2026-01-02 11:05:48.454 P00   INFO: repo1: 17-26 no archive to remove
   2026-01-02 11:05:48.455 P00   INFO: expire command end: completed successfully (103ms)

[INFO] Current backups:
stanza: identity-sau-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000018

        full backup: 20260102-085913F
            timestamp start/stop: 2026-01-02 08:59:13+00 / 2026-01-02 08:59:24+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.3MB, database backup size: 37.3MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

        full backup: 20260102-085944F
            timestamp start/stop: 2026-01-02 08:59:44+00 / 2026-01-02 08:59:49+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.3MB, database backup size: 37.3MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

    db (current)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000007

        full backup: 20260102-110513F
            timestamp start/stop: 2026-01-02 11:05:13+00 / 2026-01-02 11:05:24+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.3MB, database backup size: 37.3MB
            repo1: backup set size: 5.8MB, backup size: 5.8MB

        full backup: 20260102-110544F
            timestamp start/stop: 2026-01-02 11:05:44+00 / 2026-01-02 11:05:48+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.3MB, database backup size: 37.3MB
            repo1: backup set size: 5.8MB, backup size: 5.8MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         identity-sau-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/identity-sau-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✓[0m ✅ Backup setup completed for coordinator and all workers

[0;34m[INFO][0m Skipping 06-distribute-tables-canary.sh (test script - set RUN_TESTS=true to enable)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 07-distribute-tables.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:05:50 UTC] USER=unknown EUID=33 PID=1980591 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 11:05:50 UTC] USER=unknown EUID=33 PID=1980598 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 11:05:50 UTC] USER=unknown EUID=33 PID=1980605 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 11:05:50 UTC] USER=unknown EUID=33 PID=1980612 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS TABLE DISTRIBUTION
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════

[0;34m[INFO][0m 🔐 Secure connection established
[0;34m[INFO][0m    Host: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432
[0;34m[INFO][0m    Database: fastorder_identity_sau_main_dev_db
[0;34m[INFO][0m    SSL: verify-full (TLS 1.2+)
[0;34m[INFO][0m    Timeouts: statement=120s, idle_tx=300s

[0;34m[INFO][0m 🔍 Running preflight checks...
[0;34m[INFO][0m Testing database connectivity...
[0;32m[OK][0m   ✅ Database connection successful
[0;32m[OK][0m   ✅ Connected to correct database: fastorder_identity_sau_main_dev_db
[0;34m[INFO][0m Checking Citus extension in database fastorder_identity_sau_main_dev_db...
[0;32m[OK][0m   Citus version: 13.2-1
[0;34m[INFO][0m Checking worker registration...
[0;32m[OK][0m   Registered workers: 1
[0;34m[INFO][0m Worker nodes:
[0;34m[INFO][0m                             nodename                           | nodeport | isactive | noderole 
[0;34m[INFO][0m   -------------------------------------------------------------+----------+----------+----------
[0;34m[INFO][0m    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com |     5432 | t        | primary
[0;34m[INFO][0m   (1 row)
[0;34m[INFO][0m   

[0;34m[INFO][0m 📊 Starting table distribution...

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Distributing: auth.login_account
[0;34m[INFO][0m Description: User authentication table - distributed by region for tenant isolation
[0;34m[INFO][0m Shard key: region_hint
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 📝 Current rows: 0
[0;34m[INFO][0m Checking constraints compatibility with Citus...
[0;32m[OK][0m   ✅ No conflicting constraints found
[0;32m[OK][0m   ✅ Table already distributed - skipping
[0;34m[INFO][0m    Distribution column: region_hint
[0;32m[OK][0m   ✅ Data integrity verified (0 rows)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m[OK][0m   ✅ All tables distributed successfully!
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════

[0;34m[INFO][0m 📊 Citus Cluster Summary:

[0;34m[INFO][0m Distributed tables:
[0;34m[INFO][0m            table          |   type    | shard_key | shards | size  
[0;34m[INFO][0m   ------------------------+-----------+-----------+--------+-------
[0;34m[INFO][0m    core.tenant            | reference | <none>    |      1 | 24 kB
[0;34m[INFO][0m    core.realm             | local     | <none>    |      1 | 40 kB
[0;34m[INFO][0m    core.identity          | local     | <none>    |      1 | 72 kB
[0;34m[INFO][0m    core.device            | local     | <none>    |      1 | 48 kB
[0;34m[INFO][0m    core.identity_account  | local     | <none>    |      1 | 48 kB
[0;34m[INFO][0m    core.identity_mfa      | local     | <none>    |      1 | 40 kB
[0;34m[INFO][0m    core.external_idp_link | local     | <none>    |      1 | 48 kB
[0;34m[INFO][0m    policy.client          | local     | <none>    |      1 | 56 kB
[0;34m[INFO][0m    policy.resource        | local     | <none>    |      1 | 48 kB
[0;34m[INFO][0m    policy.scope           | local     | <none>    |      1 | 40 kB
[0;34m[INFO][0m    policy.permission      | local     | <none>    |      1 | 48 kB
[0;34m[INFO][0m    policy.role            | local     | <none>    |      1 | 56 kB
[0;34m[INFO][0m    policy.role_permission | local     | <none>    |      1 | 24 kB
[0;34m[INFO][0m    policy.identity_role   | local     | <none>    |      1 | 40 kB
[0;34m[INFO][0m    policy.policy_rule     | local     | <none>    |      1 | 48 kB
[0;34m[INFO][0m    policy.api_key         | local     | <none>    |      1 | 56 kB
[0;34m[INFO][0m    auth.login_account     | reference | <none>    |      1 | 48 kB
[0;34m[INFO][0m   (17 rows)
[0;34m[INFO][0m   

[0;34m[INFO][0m Worker capacity:
[0;34m[INFO][0m    worker | total_shards | total_size 
[0;34m[INFO][0m   --------+--------------+------------
[0;34m[INFO][0m   (0 rows)
[0;34m[INFO][0m   

[0;32m[OK][0m   Citus table distribution complete

[0;34m[INFO][0m Skipping 08-distribute-tables-rollback.sh (rollback script - run manually only)
[0;34m[INFO][0m Skipping 09-distribute-tables-test.sh (test script - set RUN_TESTS=true to enable)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 10-setup-cdc.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m CDC PIPELINE SETUP (Debezium + Elasticsearch Sink)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Log file: /var/log/fastorder/cdc/10-setup-cdc-*.log

[0;34m[INFO][0m Running CDC setup for identifier: coordinator
[2026-01-02 11:06:01] ==========================================
[2026-01-02 11:06:01] CDC SETUP SCRIPT STARTED
[2026-01-02 11:06:01] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260102_110601.log
[2026-01-02 11:06:01] ==========================================
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-02 11:06:02] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:06:02]   CDC Pipeline Setup (Debezium + ES Sink)
[2026-01-02 11:06:02] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:06:02]   Environment: identity-sau-main-dev
[2026-01-02 11:06:02]   Identifier:  coordinator
[2026-01-02 11:06:02]   Service:     identity
[2026-01-02 11:06:02] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:06:02] 📂 CDC_BASE_DIR exists: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc
[2026-01-02 11:06:02] Looking for service folder: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity
[2026-01-02 11:06:02] 
[2026-01-02 11:06:02] 📂 Found CDC configuration for service: identity
[2026-01-02 11:06:02] Scanning for subservice directories in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity
[2026-01-02 11:06:02] Found subservice: login, checking for steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps
[2026-01-02 11:06:02] 
[2026-01-02 11:06:02] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:06:02]   Setting up CDC for: identity/login
[2026-01-02 11:06:02] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 11:06:02] Found 7 step script(s) in /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps
[2026-01-02 11:06:02] 
[2026-01-02 11:06:02] 🔧 Running: 01-setup-debezium-auth-login.sh
[2026-01-02 11:06:02]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/01-setup-debezium-auth-login.sh
[2026-01-02 11:06:02]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Debezium CDC Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔍 Verifying Kafka infrastructure...
✅ db-identity-sau-main-dev-postgresql.fastorder.com resolves to 10.100.1.213
🔐 psql will use client cert for mTLS.
🔐 Retrieving credentials from secrets vault...
   Clearing cached credentials for coordinator...
✅ Credentials retrieved from secrets vault
🔐 Syncing debezium_user password in PostgreSQL...
✅ debezium_user password synchronized
🔍 Checking PostgreSQL SSL status...
✅ Server SSL is ON (verify-full + client cert).
🔧 Applying publication & grants over TLS…
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

NOTICE:  publication "cdc_pub_identity" does not exist, skipping
DROP PUBLICATION
CREATE PUBLICATION
SET
NOTICE:  Added shard table auth.login_account_102024 to publication
DO
RESET
GRANT
GRANT
GRANT
✅ Publication & grants done (including Citus shard table).
⏳ Waiting for Kafka Connect @ https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors…
[2026-01-02 11:06:11] 🔗 Waiting for Kafka Connect at: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
[2026-01-02 11:06:11] ⏳ Waiting for HTTP endpoint: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
[2026-01-02 11:06:11]    Expected codes: 200,500, timeout: 300s
[2026-01-02 11:06:11] ✅ HTTP endpoint ready: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083 (code: 200, took: 0s)
[2026-01-02 11:06:11] 🔄 Testing Connect worker readiness...
[2026-01-02 11:06:11] ✅ Kafka Connect worker ready
🧹 Cleaning up existing Debezium connector and slot (if any)...
   Step 0a: Also resetting ES Sink connector offsets (required for coordinated reset)...
   → Stopping ES Sink connector pg_identity_sau_main_dev_coordinator_es_sink...
   → Deleting ES Sink connector offsets...
   ✓ ES Sink offsets deleted successfully (HTTP 200)
   → Deleting ES Sink connector (will be recreated by 02-setup-es-sink.sh)...
   ✓ ES Sink connector cleanup complete
   Step 0b: Clearing stale Debezium connector offsets from Kafka Connect...
   → Stopping connector pg_identity_sau_main_dev_debezium_postgres...
   → Deleting connector offsets (forces fresh snapshot)...
   ✓ Connector offsets deleted successfully (HTTP 200)
   Step 1: Ensuring connector is completely removed...
   Deleting connector: pg_identity_sau_main_dev_debezium_postgres (attempt 1/10)
   ✓ Connector pg_identity_sau_main_dev_debezium_postgres does not exist (HTTP 404)
   Step 2: Waiting for replication slot to become inactive...
   ✓ Slot slot_identity_sau_main_dev does not exist (clean state)
   Step 3: Dropping replication slot...
   ✓ Slot slot_identity_sau_main_dev already dropped
   Step 4: Final verification...
✅ Cleanup complete - environment is clean for fresh CDC snapshot
🔐 Checking Debezium SSL certificate permissions...
🔍 Validating Debezium SSL certificates...
🔐 Connector will use mTLS to Postgres.
  ✓ Certificate: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt
  ✓ Key: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key
  ✓ Root CA: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
ℹ️  Skipping pre-flight connectivity test (will be validated by Kafka Connect)
📤 Upserting connector: PUT https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_identity_sau_main_dev_debezium_postgres/config
   Attempt 1/5: Sending PUT request to Kafka Connect...
   (This may take up to 60s as Connect validates the configuration)
   ✅ Success (HTTP 201)

🌐 HTTP Response: 201
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Response body:
{
  "name": "pg_identity_sau_main_dev_debezium_postgres",
  "config": {
    "name": "pg_identity_sau_main_dev_debezium_postgres",
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "plugin.name": "pgoutput",
    "database.hostname": "db-identity-sau-main-dev-postgresql.fastorder.com",
    "database.port": "5432",
    "database.dbname": "fastorder_identity_sau_main_dev_db",
    "database.user": "debezium_user",
    "database.password": "tvXyu5jP8RRTy0dSBWaj5uS5f",
    "database.sslmode": "verify-full",
    "database.sslrootcert": "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt",
    "database.sslcert": "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt",
    "database.sslkey": "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key",
    "publication.name": "cdc_pub_identity",
    "publication.autocreate.mode": "disabled",
    "slot.name": "slot_identity_sau_main_dev",
    "topic.prefix": "identity_sau_main_dev_cdc",
    "schema.include.list": "auth",
    "table.include.list": "auth.login_account,auth.login_account_[0-9]+",
    "transforms": "unwrap,route",
    "transforms.unwrap.add.fields": "op,ts_ms",
    "transforms.unwrap.delete.handling.mode": "rewrite",
    "transforms.unwrap.drop.tombstones": "false",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
    "transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.route.regex": "^identity_sau_main_dev_cdc\\.auth\\.login_account(_[0-9]+)?$",
    "transforms.route.replacement": "identity_sau_main_dev_account_router",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",
    "snapshot.mode": "always"
  },
  "tasks": [],
  "type": "source"
}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Connector upserted.
🔄 Verifying connector task startup...
✅ Debezium connector task is RUNNING
ℹ️  Source table auth.login_account has 0 rows.
ℹ️  Snapshot will be metadata-only; offsets may stay empty until first change.
⏳ Waiting for Debezium initial snapshot to complete...
   📊 Slot status: restart_lsn=0/8015DE8, confirmed_flush_lsn=0/8015E20
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (0s elapsed)
   ⏳ Snapshot in progress... (5s elapsed)
   ⏳ Snapshot in progress... (10s elapsed)
   📊 Slot status: restart_lsn=0/8015DE8, confirmed_flush_lsn=0/8015E20
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (15s elapsed)
   ⏳ Snapshot in progress... (20s elapsed)
   ⏳ Snapshot in progress... (25s elapsed)
   📊 Slot status: restart_lsn=0/8015DE8, confirmed_flush_lsn=0/8015E20
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (30s elapsed)
   ⏳ Snapshot in progress... (35s elapsed)
   ⏳ Snapshot in progress... (40s elapsed)
   📊 Slot status: restart_lsn=0/8015DE8, confirmed_flush_lsn=0/8015E20
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (45s elapsed)
   ⏳ Snapshot in progress... (50s elapsed)
   ⏳ Snapshot in progress... (55s elapsed)
   📊 Slot status: restart_lsn=0/8015DE8, confirmed_flush_lsn=0/8015E20
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (60s elapsed)
   ⏳ Snapshot in progress... (65s elapsed)
   ⏳ Snapshot in progress... (70s elapsed)
   📊 Slot status: restart_lsn=0/8015DE8, confirmed_flush_lsn=0/8015E20
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (75s elapsed)
   ⏳ Snapshot in progress... (80s elapsed)
   ⏳ Snapshot in progress... (85s elapsed)
   📊 Slot status: restart_lsn=0/8015DE8, confirmed_flush_lsn=0/8015E20
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (90s elapsed)
   ⏳ Snapshot in progress... (95s elapsed)
   ⏳ Snapshot in progress... (100s elapsed)
   📊 Slot status: restart_lsn=0/8015DE8, confirmed_flush_lsn=0/8015E20
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (105s elapsed)
   ⏳ Snapshot in progress... (110s elapsed)
   ⏳ Snapshot in progress... (115s elapsed)

⚠️  WARNING: Snapshot wait timeout (120s) on EMPTY table.
   Offsets are still empty, but source table has 0 rows.
   Proceeding anyway – CDC health will be verified by test inserts.

✅ Debezium connector is RUNNING after snapshot
🔍 Final verification: Checking Debezium offsets are recorded...
   ℹ️  Source table auth.login_account has 0 rows
   ℹ️  Skipping offset verification (no data to snapshot)
✅ Debezium connector verified RUNNING (empty source table)
🔄 Phase 2: Updating connector to snapshot.mode=initial...
✅ Connector updated to snapshot.mode=initial (HTTP 200)
✅ Connector verified RUNNING after Phase 2 update
✅ Debezium connector configured successfully (two-phase snapshot complete)
[2026-01-02 11:08:43] ✅ Completed: 01-setup-debezium-auth-login.sh
[2026-01-02 11:08:44] 
[2026-01-02 11:08:44] 🔧 Running: 02-setup-es-sink.sh
[2026-01-02 11:08:44]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/02-setup-es-sink.sh
[2026-01-02 11:08:44]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[WARN] Master/coordinator not found, using node-01
[INFO] Using ES domain: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com
🔐 Retrieving keystore passwords from secrets manager...
[INFO] Retrieving Kafka truststore password...
[0;32m✅ Retrieved passwords from remote backend[0m
✅ Retrieved Kafka truststore password
[INFO] Retrieving Elasticsearch P12 password...
[0;34m[INFO][0m 🔍 Checking secrets backend (provider: aws)...
[0;32m✅ Retrieved passwords from remote backend[0m
[0;34m[INFO][0m ✅ Using existing passwords from backend
✅ Retrieved/generated Elasticsearch P12 password
✅ Keystore passwords retrieved successfully
   - Kafka truststore password: yOb0eqkA... (32 chars)
   - ES P12 password: 8siDJx7z... (32 chars)
[INFO] 🔐 Clearing cached ES credentials to ensure fresh retrieval...
[0;34m[INFO][0m [INFO] ✅ Using ES password from centralized secrets vault (identifier: node-01)
[INFO] 🔐 Verifying Elasticsearch accepts client certificate...
[INFO] ✅ Elasticsearch accepting client certificate
[INFO] 🔐 Setting up ES client keystore using Kafka client certificate...
[INFO]    Certificate: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem (signed by Fastorder RA Root CA)
[INFO] 📋 Creating ES client P12 keystore from Kafka client certificate...
[2026-01-02 11:08:55 UTC] USER=www-data EUID=0 PID=1983730 ACTION=fsop ARGS=mv /tmp/es-client-1983520.p12 /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[2026-01-02 11:08:55 UTC] USER=www-data EUID=0 PID=1983739 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[2026-01-02 11:08:55 UTC] USER=www-data EUID=0 PID=1983749 ACTION=fsop ARGS=chmod 600 /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[INFO] ✅ Created ES client keystore: /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[INFO]    Using Kafka client cert signed by Fastorder RA Root CA
[INFO] ℹ️  Using Kafka truststore and adding ES CA certificate
[2026-01-02 11:08:55 UTC] USER=www-data EUID=0 PID=1983758 ACTION=fsop ARGS=test -f /opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks
[INFO] ✅ ES CA already in truststore
[0;34m[INFO][0m [INFO] 🔗 Waiting for Kafka Connect at: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
[0;34m[INFO][0m [INFO] ✅ Connect HTTP ready (code 200)
[0;34m[INFO][0m [INFO] 🔍 Verifying Debezium connector snapshot status...
[0;34m[INFO][0m [INFO] ℹ️  Source table auth.login_account has 0 rows.
[0;34m[INFO][0m [INFO]    Skipping Debezium snapshot wait (metadata-only snapshot on empty table).
[0;34m[INFO][0m [INFO] 🔌 Cleaning up existing ES Sink connector: pg_identity_sau_main_dev_coordinator_es_sink
[0;34m[INFO][0m [INFO]    → Deleting connector...
[0;34m[INFO][0m [INFO]    HTTP 404 (404 is fine)
[0;34m[INFO][0m [INFO] 🔐 Validating Elasticsearch credentials...
[0;34m[INFO][0m [INFO] ✅ ES credentials validated successfully
[0;34m[INFO][0m [INFO] 🔧 Creating required Elasticsearch ingest pipelines: identity-embed-pipeline-001
[0;34m[INFO][0m [INFO] ✅ Pipeline identity-embed-pipeline-001 created successfully
[0;34m[INFO][0m [INFO] 🔧 Ensuring CDC index has no default_pipeline requirement...
[0;34m[INFO][0m [INFO] ✅ Removed default_pipeline from index (if any)
[0;34m[INFO][0m [INFO] 🔧 Ensuring dynamic mapping is enabled...
[0;34m[INFO][0m [INFO] ✅ Dynamic mapping enabled for identity_sau_main_dev_account_router
[DEBUG] ES_TRUSTSTORE=/opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks
[DEBUG] ES_CLIENT_P12=/opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[DEBUG] TRUSTSTORE_PASS=yOb0eqkA...
[DEBUG] P12_PASS=8siDJx7z...
== Outgoing connector config (snippet) ==
2:  "name": "pg_identity_sau_main_dev_coordinator_es_sink",
6:  "connection.url": "https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200",
19:  "index": "identity_sau_main_dev_account_router",
[INFO] ⚠️  Skipping pre-validation - will validate on PUT...
[0;34m[INFO][0m [INFO] ✅ Proceeding to PUT
[2026-01-02 11:08:58] [1/3] Upserting connector via PUT https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_identity_sau_main_dev_coordinator_es_sink/config
🌐 HTTP 201
✅ Connector created/updated successfully
{
  "name": "pg_identity_sau_main_dev_coordinator_es_sink",
  "config": {
    "name": "pg_identity_sau_main_dev_coordinator_es_sink",
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "tasks.max": "1",
    "topics": "identity_sau_main_dev_account_router",
    "connection.url": "https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200",
    "elastic.security.protocol": "SSL",
    "elastic.https.ssl.hostname.verification": "true",
    "elastic.https.ssl.truststore.location": "/opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks",
    "elastic.https.ssl.truststore.password": "yOb0eqkAqtj8HEWebgA7nf04YlqsLw44",
    "elastic.https.ssl.truststore.type": "JKS",
    "elastic.https.ssl.keystore.location": "/opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12",
    "elastic.https.ssl.keystore.password": "8siDJx7zdDhhu5iMMZwnhZfTaGFSgCvh",
    "elastic.https.ssl.keystore.type": "PKCS12",
    "elastic.username": "elastic",
    "elastic.password": "T+kMy0e84aGeV204NzYK",
    "connection.username": "elastic",
    "connection.password": "T+kMy0e84aGeV204NzYK",
    "index": "identity_sau_main_dev_account_router",
    "key.ignore": "true",
    "schema.ignore": "true",
    "behavior.on.null.values": "delete",
    "write.method": "upsert",
    "type.name": "_doc",
    "max.in.flight.requests": "1",
    "batch.size": "2000",
    "linger.ms": "100",
    "flush.timeout.ms": "60000",
    "max.retries": "10",
    "retry.backoff.ms": "5000",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false"
  },
  "tasks": [],
  "type": "sink"
}
{
  "pg_identity_sau_main_dev_debezium_postgres": {
    "status": {
      "name": "pg_identity_sau_main_dev_debezium_postgres",
      "connector": {
        "state": "RUNNING",
        "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
      },
      "tasks": [
        {
          "id": 0,
          "state": "RUNNING",
          "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
        }
      ],
      "type": "source"
    }
  },
  "pg_identity_sau_to_universe_main_dev_es_sink": {
    "status": {
      "name": "pg_identity_sau_to_universe_main_dev_es_sink",
      "connector": {
        "state": "RUNNING",
        "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
      },
      "tasks": [
        {
          "id": 0,
          "state": "RUNNING",
          "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
        }
      ],
      "type": "sink"
    }
  },
  "pg_identity_sau_main_dev_coordinator_es_sink": {
    "status": {
      "name": "pg_identity_sau_main_dev_coordinator_es_sink",
      "connector": {
        "state": "RUNNING",
        "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
      },
      "tasks": [],
      "type": "sink"
    }
  }
}
[0;34m[INFO][0m [INFO] 🔗 Creating ES alias for application compatibility...
[0;34m[INFO][0m [INFO]    ⏳ Waiting for ES index to be created... (0s)
[0;34m[INFO][0m [INFO]    ⏳ Waiting for ES index to be created... (5s)
[0;34m[INFO][0m [INFO]    ⏳ Waiting for ES index to be created... (10s)
[0;34m[INFO][0m [INFO]    ⏳ Waiting for ES index to be created... (15s)
[0;34m[INFO][0m [INFO]    ⏳ Waiting for ES index to be created... (20s)
[0;34m[INFO][0m [INFO]    ⏳ Waiting for ES index to be created... (25s)
[0;34m[INFO][0m [INFO]    ⏳ Waiting for ES index to be created... (30s)
[0;34m[INFO][0m [INFO]    ⏳ Waiting for ES index to be created... (35s)
[0;34m[INFO][0m [INFO]    ⏳ Waiting for ES index to be created... (40s)
[0;34m[INFO][0m [INFO]    ⏳ Waiting for ES index to be created... (45s)
[0;34m[INFO][0m [INFO]    ⏳ Waiting for ES index to be created... (50s)
[0;34m[INFO][0m [INFO]    ⏳ Waiting for ES index to be created... (55s)
[0;33m[WARN] ⚠️  ES index not created within 60s, skipping alias creation[0m

🔍 Final verification: Checking ES document count...
   PostgreSQL auth.login_account: 0 rows
ℹ️  PostgreSQL table is empty - skipping ES verification
✅ Done.
[2026-01-02 11:10:02] ✅ Completed: 02-setup-es-sink.sh
[2026-01-02 11:10:02] 
[2026-01-02 11:10:02] 🔧 Running: 03-setup-es-universe-sink.sh
[2026-01-02 11:10:02]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/03-setup-es-universe-sink.sh
[2026-01-02 11:10:02]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Universe Identity ES Sink Setup (Dual-Sink Pattern)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Source Zone:  sau
  Connector:      pg_identity_sau_to_universe_main_dev_es_sink
  Source Topic:   identity_sau_main_dev_account_router
  Universe ES:      search-identity-universe-main-dev.fastorder.com:9200
  Universe Index:   identity_universe_main_dev_account_router
  Zone Field:   zone: "sau" (added to each document)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Retrieving keystore passwords from secrets manager...
[0;32m✅ Retrieved passwords from remote backend[0m
✅ Retrieved Kafka truststore password
[0;34m[INFO][0m 🔍 Checking secrets backend (provider: aws)...
[0;32m✅ Retrieved passwords from remote backend[0m
[0;34m[INFO][0m ✅ Using existing passwords from backend
✅ Retrieved/generated Elasticsearch P12 password
🔐 Retrieving Universe ES password...
[0;34m[INFO][0m [INFO] ✅ Retrieved Universe ES password from vault (identifier: node-01)
❌ missing CA file: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-02 11:10:22] ❌ FAILED: 03-setup-es-universe-sink.sh (exit code: 1)
[2026-01-02 11:10:22] ❌ CRITICAL: This is a required step for CDC pipeline. Aborting.

[0;31m[ERROR][0m ❌ Database infrastructure (postgresql) setup failed with exit code: 1

📥 Download Full Logs

06-finalizing local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

Total Steps

Succeeded

Failed

Running

Pending

30 minutes

Total Steps Time

← Back to Dashboard 🔍 View Environment