📊 Provisioning Job Status

Environment: User Sau Main Dev on web-03

❌ Failed

⏱️ Timing Summary

🕐

Requested 2026-01-19 13:27:09 2 weeks ago

▶️

Started 2026-01-19 13:27:10 2 weeks ago

🏁

Finished 2026-01-19 13:37:12 2 weeks ago

⏲️

Total Duration 10 minutes

📋 Job Details

Job ID: 01c92b49-cb3d-406f-92b1-bfc5cb7c45c4

Action: SETUP

Status: ❌ FAILED

Environment: user-sau-main-dev

Resource: web-03 (Provider)

Requested By: admin

Parameters:

"{\"env\": \"dev\", \"zone\": \"sau\", \"branch\": \"main\", \"db_app\": \"postgresql\", \"service\": \"user\", \"es_nodes\": 1, \"db_enabled\": true, \"pg_standby\": 1, \"pg_workers\": 1, \"search_app\": \"elasticsearch\", \"description\": \"\", \"iam_enabled\": false, \"worker_1_ip\": \"10.100.1.42\", \"eventbus_app\": \"kafka\", \"es_https_mode\": \"direct\", \"service_es_ip\": \"10.100.1.4\", \"worker_1_fqdn\": \"db-user-sau-main-dev-postgresql-worker-01.fastorder.com\", \"search_enabled\": true, \"service_app_ip\": \"10.100.1.2\", \"service_obs_ip\": \"10.100.1.18\", \"service_es_fqdn\": \"search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com\", \"service_otlp_ip\": \"10.100.1.30\", \"eventbus_enabled\": true, \"service_app_fqdn\": \"app-user-sau-main-dev.fastorder.com\", \"service_audit_ip\": \"10.100.1.32\", \"service_obs_fqdn\": \"obs-user-sau-main-dev.fastorder.com\", \"service_tempo_ip\": \"10.100.1.28\", \"service_endpoints\": \"[{\\\"ip\\\":\\\"10.100.1.3\\\",\\\"fqdn\\\":\\\"app-user-sau-main-dev.fastorder.com\\\",\\\"service\\\":\\\"app\\\"},{\\\"ip\\\":\\\"10.100.1.5\\\",\\\"fqdn\\\":\\\"search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com\\\",\\\"service\\\":\\\"es_coordinator\\\"},{\\\"ip\\\":\\\"10.100.1.7\\\",\\\"fqdn\\\":\\\"search-user-sau-main-dev-elasticsearch-node-01.fastorder.com\\\",\\\"service\\\":\\\"es_node_1\\\"},{\\\"ip\\\":\\\"10.100.1.9\\\",\\\"fqdn\\\":\\\"eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com\\\",\\\"service\\\":\\\"kafka_broker_1\\\"},{\\\"ip\\\":\\\"10.100.1.11\\\",\\\"fqdn\\\":\\\"eventbus-user-sau-main-dev-kafka-connect.fastorder.com\\\",\\\"service\\\":\\\"kafka_connect\\\"},{\\\"ip\\\":\\\"10.100.1.13\\\",\\\"fqdn\\\":\\\"schema-user-sau-main-dev-kafka-registry.fastorder.com\\\",\\\"service\\\":\\\"kafka_registry\\\"},{\\\"ip\\\":\\\"10.100.1.15\\\",\\\"fqdn\\\":\\\"db-user-sau-main-dev-postgresql-coordinator.fastorder.com\\\",\\\"service\\\":\\\"pg_coordinator\\\"},{\\\"ip\\\":\\\"10.100.1.17\\\",\\\"fqdn\\\":\\\"db-user-sau-main-dev-postgresql-bouncer.fastorder.com\\\",\\\"service\\\":\\\"pgbouncer\\\"},{\\\"ip\\\":\\\"10.100.1.19\\\",\\\"fqdn\\\":\\\"obs-user-sau-main-dev.fastorder.com\\\",\\\"service\\\":\\\"obs\\\"},{\\\"ip\\\":\\\"10.100.1.21\\\",\\\"fqdn\\\":\\\"metrics-user-sau-main-dev-prometheus.fastorder.com\\\",\\\"service\\\":\\\"metrics\\\"},{\\\"ip\\\":\\\"10.100.1.23\\\",\\\"fqdn\\\":\\\"dashboards-user-sau-main-dev-grafana.fastorder.com\\\",\\\"service\\\":\\\"dashboards\\\"},{\\\"ip\\\":\\\"10.100.1.25\\\",\\\"fqdn\\\":\\\"alerts-user-sau-main-dev-alertmanager.fastorder.com\\\",\\\"service\\\":\\\"alerts\\\"},{\\\"ip\\\":\\\"10.100.1.27\\\",\\\"fqdn\\\":\\\"logstore-user-sau-main-dev-clickhouse.fastorder.com\\\",\\\"service\\\":\\\"logs\\\"},{\\\"ip\\\":\\\"10.100.1.29\\\",\\\"fqdn\\\":\\\"traces-user-sau-main-dev-tempo.fastorder.com\\\",\\\"service\\\":\\\"traces\\\"},{\\\"ip\\\":\\\"10.100.1.31\\\",\\\"fqdn\\\":\\\"telemetry-user-sau-main-dev-opentelemetry.fastorder.com\\\",\\\"service\\\":\\\"telemetry\\\"},{\\\"ip\\\":\\\"10.100.1.33\\\",\\\"fqdn\\\":\\\"audit-user-sau-main-dev.fastorder.com\\\",\\\"service\\\":\\\"audit\\\"},{\\\"ip\\\":\\\"10.100.1.35\\\",\\\"fqdn\\\":\\\"backup-user-sau-main-dev-db-postgresql.fastorder.com\\\",\\\"service\\\":\\\"backup_pg\\\"},{\\\"ip\\\":\\\"10.100.1.37\\\",\\\"fqdn\\\":\\\"backup-user-sau-main-dev-eventbus-kafka.fastorder.com\\\",\\\"service\\\":\\\"backup_kafka\\\"},{\\\"ip\\\":\\\"10.100.1.39\\\",\\\"fqdn\\\":\\\"backup-user-sau-main-dev-search-elasticsearch.fastorder.com\\\",\\\"service\\\":\\\"backup_es\\\"},{\\\"ip\\\":\\\"10.100.1.41\\\",\\\"fqdn\\\":\\\"backup-user-sau-main-dev-orchestrator.fastorder.com\\\",\\\"service\\\":\\\"backup_orchestrator\\\"}]\", \"service_otlp_fqdn\": \"telemetry-user-sau-main-dev-opentelemetry.fastorder.com\", \"postgresql_enabled\": true, \"service_audit_fqdn\": \"audit-user-sau-main-dev.fastorder.com\", \"service_grafana_ip\": \"10.100.1.22\", \"service_tempo_fqdn\": \"traces-user-sau-main-dev-tempo.fastorder.com\", \"service_backup_es_ip\": \"10.100.1.38\", \"service_backup_pg_ip\": \"10.100.1.34\", \"service_es_node_1_ip\": \"10.100.1.6\", \"service_grafana_fqdn\": \"dashboards-user-sau-main-dev-grafana.fastorder.com\", \"service_pgbouncer_ip\": \"10.100.1.16\", \"service_prometheus_ip\": \"10.100.1.20\", \"worker_1_standby_1_ip\": \"10.100.1.43\", \"service_backup_es_fqdn\": \"backup-user-sau-main-dev-search-elasticsearch.fastorder.com\", \"service_backup_pg_fqdn\": \"backup-user-sau-main-dev-db-postgresql.fastorder.com\", \"service_es_node_1_fqdn\": \"search-user-sau-main-dev-elasticsearch-node-01.fastorder.com\", \"service_log_backend_ip\": \"10.100.1.26\", \"service_pgbouncer_fqdn\": \"db-user-sau-main-dev-postgresql-bouncer.fastorder.com\", \"service_alertmanager_ip\": \"10.100.1.24\", \"service_backup_kafka_ip\": \"10.100.1.36\", \"service_prometheus_fqdn\": \"metrics-user-sau-main-dev-prometheus.fastorder.com\", \"worker_1_standby_1_fqdn\": \"db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com\", \"service_kafka_connect_ip\": \"10.100.1.10\", \"service_log_backend_fqdn\": \"logstore-user-sau-main-dev-clickhouse.fastorder.com\", \"service_alertmanager_fqdn\": \"alerts-user-sau-main-dev-alertmanager.fastorder.com\", \"service_backup_kafka_fqdn\": \"backup-user-sau-main-dev-eventbus-kafka.fastorder.com\", \"service_kafka_broker_1_ip\": \"10.100.1.8\", \"service_kafka_registry_ip\": \"10.100.1.12\", \"service_pg_coordinator_ip\": \"10.100.1.14\", \"service_kafka_connect_fqdn\": \"eventbus-user-sau-main-dev-kafka-connect.fastorder.com\", \"postgresql_run_verification\": true, \"service_kafka_broker_1_fqdn\": \"eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com\", \"service_kafka_registry_fqdn\": \"schema-user-sau-main-dev-kafka-registry.fastorder.com\", \"service_pg_coordinator_fqdn\": \"db-user-sau-main-dev-postgresql-coordinator.fastorder.com\", \"service_backup_orchestrator_ip\": \"10.100.1.40\", \"service_backup_orchestrator_fqdn\": \"backup-user-sau-main-dev-orchestrator.fastorder.com\"}"

❌ Error: One or more steps failed. Check run logs for details.

⚠️ Job Failed

This job encountered an error. You can restart from the failed step.

📢 Viewing Old Job Attempt

This job has been restarted. You are viewing an older attempt. The logs and status shown below are from the latest retry.

🔄 View Latest Attempt Latest Status:

🔄 Resume & Restart Options

This job failed at one of the steps below. You can resume from where it failed to save time and avoid re-running successful steps.

💡

1 step failed

📝 Execution Steps (9)

0/9 completed 1 failed

0% (0/9 steps)

00-preflight-checks local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

00-terraform-provision local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

01-prepare-environment local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

02-iam local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

02-observability-cell local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

03-search local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

04-eventbus local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

05-db local

❌ FAILED

⏰ Started: 2026-01-19 13:27:10

🏁 Finished: 2026-01-19 13:37:12

⏱️ Duration: 10 minutes

📄 View Logs (529491 chars)

[0;34m[INFO][0m Using database engine from DB_ENGINE environment variable: postgresql
[0;34m[INFO][0m Cleaning up any existing locks...

[0;32m[1mStarting database engine: postgresql[0m
[1;33m═══════════════════════════════════════════════[0m

[0;34m[INFO][0m Using environment from web interface: user-sau-main-dev
[0;32m[2026-01-19 13:27:10][0m Using web-provided environment: user-sau-main-dev
[0;32m[2026-01-19 13:27:10][0m Service: user, Zone: sau, Branch: main, Env: dev
[0;32m✓[0m Environment initialized successfully (mode: general)
[0;34m[INFO][0m Checking observability cell readiness: obs-user-sau-main-dev
[1;32m[OK][0m   Observability cell endpoints registered for user-sau-main-dev
[0;34m[INFO][0m Observability cell verified for user-sau-main-dev
[0;34m[INFO][0m Monitoring will be configured after PostgreSQL deployment (step 10-monitoring-setup.sh)
[0;34m[INFO][0m Citus mode ENABLED
[0;34m[INFO][0m → Coordinator + 1 worker(s) + 1 standby node(s) per worker
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up coordinator (Citus control plane)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-19 13:27:11 UTC] USER=unknown EUID=33 PID=1307653 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-19 13:27:11 UTC] USER=unknown EUID=33 PID=1307663 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-19 13:27:11 UTC] USER=unknown EUID=33 PID=1307672 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-19 13:27:11 UTC] USER=unknown EUID=33 PID=1307680 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-19 13:27:11 UTC] USER=unknown EUID=33 PID=1307688 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-19 13:27:11 UTC] USER=unknown EUID=33 PID=1307695 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for user in sau-dev...
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4 JOB_UUID=01c92b49-cb3d-406f-92b1-bfc5cb7c45c4

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.231
[0;34m[INFO][0m Primary hostname: db-user-sau-main-dev-postgresql-coordinator.fastorder.com

[0;34m[INFO][0m Adding /etc/hosts entries for coordinator...
[0;34m[INFO][0m   1. db-user-sau-main-dev-postgresql.fastorder.com → 10.100.1.231 (primary/short)
[0;34m[INFO][0m   2. db-user-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.231 (compatibility)

[0;34m[INFO][0m   ✅ db-user-sau-main-dev-postgresql.fastorder.com already exists with correct IP
[0;34m[INFO][0m   ✅ db-user-sau-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;32m✅[0m   ✅ Network & DNS configuration complete
[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
  10.100.1.231    db-user-sau-main-dev-postgresql-coordinator.fastorder.com
  10.100.1.231    db-user-sau-main-dev-postgresql.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-user-sau-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      user-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-user-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-19 13:27:14 UTC] USER=www-data EUID=0 PID=1307905 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator and /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:27:14 UTC] USER=www-data EUID=0 PID=1307915 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-19 13:27:15 UTC] USER=www-data EUID=0 PID=1307925 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1307860
[2026-01-19 13:27:15 UTC] USER=www-data EUID=0 PID=1307934 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1307860/ra_root.crt
[2026-01-19 13:27:15 UTC] USER=www-data EUID=0 PID=1307945 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1307860/ra_root.key
[2026-01-19 13:27:15 UTC] USER=www-data EUID=0 PID=1307954 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1307860/ra_root.crt
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-19 13:27:18 UTC] USER=www-data EUID=0 PID=1308069 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1307860/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-19 13:27:18 UTC] USER=www-data EUID=0 PID=1308078 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1307860/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-19 13:27:18 UTC] USER=www-data EUID=0 PID=1308088 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-19 13:27:18 UTC] USER=www-data EUID=0 PID=1308097 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1307860/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:27:19 UTC] USER=www-data EUID=0 PID=1308116 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-19 13:27:19 UTC] USER=www-data EUID=0 PID=1308151 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-19 13:27:19 UTC] USER=www-data EUID=0 PID=1308160 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-19 13:27:19 UTC] USER=www-data EUID=0 PID=1308169 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-19 13:27:19 UTC] USER=www-data EUID=0 PID=1308178 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-19 13:27:19 UTC] USER=www-data EUID=0 PID=1308187 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:27:19 UTC] USER=www-data EUID=0 PID=1308198 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-user-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:user-sau-main-dev.fastorder.com, DNS:db-user-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:db-user-sau-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-user-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-user-sau-main-dev-postgresql.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: user-sau-main-dev
Node:        coordinator
Primary CN:  db-user-sau-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-user-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-user-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308278 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308289 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308298 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308307 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308322 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308331 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308340 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308358 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308376 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308385 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308394 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308403 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308412 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308421 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308430 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308439 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308448 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308458 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308467 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308476 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:20 UTC] USER=www-data EUID=0 PID=1308521 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308530 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308539 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308548 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308558 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308567 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308576 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308586 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308597 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308607 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308617 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308627 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308636 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308645 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308654 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308663 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308672 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308681 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308690 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308699 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308708 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308718 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308727 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308737 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308747 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308756 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308766 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308776 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308785 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308794 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308803 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308812 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308821 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308830 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308839 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:27:21 UTC] USER=www-data EUID=0 PID=1308848 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1308858 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1308868 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1308877 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1308886 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1308895 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1308905 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1308914 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1308925 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1308938 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1308951 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1308960 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1308969 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1308978 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1308988 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1308998 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1309007 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1309016 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1309025 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1309034 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1309043 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1309052 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1309061 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:27:22 UTC] USER=www-data EUID=0 PID=1309070 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: postgres
Node: coordinator
FQDN: db-user-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-user-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:27:23 UTC] USER=www-data EUID=0 PID=1309142 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-19 13:27:23 UTC] USER=www-data EUID=0 PID=1309161 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-19 13:27:23 UTC] USER=www-data EUID=0 PID=1309172 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309190 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309217 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309227 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309237 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309246 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309264 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309273 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309282 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309291 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309309 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309318 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309327 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309336 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309345 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309354 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309372 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309398 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309407 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309416 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309425 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309452 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309461 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309470 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309479 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:27:24 UTC] USER=www-data EUID=0 PID=1309488 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309497 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309507 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309519 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309537 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309546 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309555 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309564 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309573 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309593 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309610 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309619 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309628 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309638 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309648 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309657 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309666 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309675 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309684 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309693 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309702 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309711 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309720 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309729 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309738 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309747 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309757 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309769 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309778 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309787 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:27:25 UTC] USER=www-data EUID=0 PID=1309797 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:27:26 UTC] USER=www-data EUID=0 PID=1309843 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:27:26 UTC] USER=www-data EUID=0 PID=1309852 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-19 13:27:26 UTC] USER=www-data EUID=0 PID=1309861 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-19 13:27:26 UTC] USER=www-data EUID=0 PID=1309871 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-19 13:27:26 UTC] USER=www-data EUID=0 PID=1309881 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:27:26 UTC] USER=www-data EUID=0 PID=1309898 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:27:26 UTC] USER=www-data EUID=0 PID=1309909 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
   ✅ Symlinked ca.pem
[2026-01-19 13:27:26 UTC] USER=www-data EUID=0 PID=1309929 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:27:26 UTC] USER=www-data EUID=0 PID=1309938 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:27:26 UTC] USER=www-data EUID=0 PID=1309947 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:27:26 UTC] USER=www-data EUID=0 PID=1309956 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:27:26 UTC] USER=www-data EUID=0 PID=1309966 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:27:26 UTC] USER=www-data EUID=0 PID=1309975 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: postgres
Node: coordinator
FQDN: db-user-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 📦 02 setup pg instance...
[0;34m[DEADLOCK-PREVENTION][0m Deadlock prevention library loaded
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-coordinator-postgresql environment: db-user-sau-main-dev-postgresql-coordinator.fastorder.com (10.100.1.231)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.231
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m Data dir:   /data/postgresql/17/user-sau-main-dev/coordinator
[0;34m[INFO][0m Port:       5432
[0;34m[INFO][0m Hostname:   db-user-sau-main-dev-postgresql-coordinator
[2026-01-19 13:27:28 UTC] USER=www-data EUID=0 PID=1310101 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:27:28 UTC] USER=www-data EUID=0 PID=1310122 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:27:28 UTC] USER=www-data EUID=0 PID=1310143 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:27:28 UTC] USER=www-data EUID=0 PID=1310166 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[1;33m[WARN][0m Server certificate not found at /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[0;34m[INFO][0m Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-user-sau-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      user-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-user-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-19 13:27:28 UTC] USER=www-data EUID=0 PID=1310219 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator and /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:27:28 UTC] USER=www-data EUID=0 PID=1310230 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-19 13:27:28 UTC] USER=www-data EUID=0 PID=1310240 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1310173
[2026-01-19 13:27:28 UTC] USER=www-data EUID=0 PID=1310249 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1310173/ra_root.crt
[2026-01-19 13:27:28 UTC] USER=www-data EUID=0 PID=1310258 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1310173/ra_root.key
[2026-01-19 13:27:28 UTC] USER=www-data EUID=0 PID=1310267 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1310173/ra_root.crt
[2026-01-19 13:27:29 UTC] USER=www-data EUID=0 PID=1310276 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1310173/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-19 13:27:32 UTC] USER=www-data EUID=0 PID=1310420 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1310173/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-19 13:27:32 UTC] USER=www-data EUID=0 PID=1310429 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-19 13:27:32 UTC] USER=www-data EUID=0 PID=1310438 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1310173/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:27:32 UTC] USER=www-data EUID=0 PID=1310451 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:27:32 UTC] USER=www-data EUID=0 PID=1310465 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:27:32 UTC] USER=www-data EUID=0 PID=1310480 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-19 13:27:32 UTC] USER=www-data EUID=0 PID=1310492 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-19 13:27:32 UTC] USER=www-data EUID=0 PID=1310501 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-19 13:27:32 UTC] USER=www-data EUID=0 PID=1310511 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-19 13:27:33 UTC] USER=www-data EUID=0 PID=1310520 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-19 13:27:33 UTC] USER=www-data EUID=0 PID=1310536 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: user-sau-main-dev
Node:        coordinator
Primary CN:  db-user-sau-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-user-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-19 13:27:33 UTC] USER=www-data EUID=0 PID=1310590 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-19 13:27:33 UTC] USER=www-data EUID=0 PID=1310599 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-19 13:27:33 UTC] USER=www-data EUID=0 PID=1310613 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
[0;32m[OK][0m   mTLS certificates OK (server cert + client certs verified) and keys secured
[0;34m[INFO][0m Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-19 13:27:33 UTC] USER=www-data EUID=0 PID=1310640 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-coordinator.service
[2026-01-19 13:27:33 UTC] USER=www-data EUID=0 PID=1310667 ACTION=passthru ARGS=systemctl stop postgresql
[1;33m[WARN][0m Cleaning stale socket directory /var/run/postgresql-user-sau-main-dev-coordinator
[2026-01-19 13:27:33 UTC] USER=www-data EUID=0 PID=1310701 ACTION=fsop ARGS=rm -rf /var/run/postgresql-user-sau-main-dev-coordinator
[0;32m[OK][0m   No conflicting Postgres left on port 5432
[0;32m[OK][0m   Using postgres password from vault provider
[2026-01-19 13:27:35 UTC] USER=www-data EUID=0 PID=1310808 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.u11y6p
[2026-01-19 13:27:36 UTC] USER=www-data EUID=0 PID=1310829 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.u11y6p
[2026-01-19 13:27:36 UTC] USER=www-data EUID=0 PID=1310852 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/user-sau-main-dev
[2026-01-19 13:27:36 UTC] USER=www-data EUID=0 PID=1310874 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev
[2026-01-19 13:27:36 UTC] USER=www-data EUID=0 PID=1310899 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/user-sau-main-dev
[0;34m[INFO][0m Initializing cluster in /data/postgresql/17/user-sau-main-dev/coordinator (SCRAM; pwfile)
[1;33m[WARN][0m Removing existing data directory: /data/postgresql/17/user-sau-main-dev/coordinator
[2026-01-19 13:27:36 UTC] USER=www-data EUID=0 PID=1310920 ACTION=fsop ARGS=rm -rf /data/postgresql/17/user-sau-main-dev/coordinator
[2026-01-19 13:27:36 UTC] USER=www-data EUID=0 PID=1310966 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/user-sau-main-dev/coordinator
[2026-01-19 13:27:37 UTC] USER=www-data EUID=0 PID=1310988 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev/coordinator
[2026-01-19 13:27:37 UTC] USER=www-data EUID=0 PID=1311010 ACTION=fsop ARGS=chmod 700 /data/postgresql/17/user-sau-main-dev/coordinator
[2026-01-19 13:27:37 UTC] USER=www-data EUID=0 PID=1311031 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-user-sau-main-dev-coordinator
[2026-01-19 13:27:37 UTC] USER=www-data EUID=0 PID=1311052 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-user-sau-main-dev-coordinator
[2026-01-19 13:27:37 UTC] USER=www-data EUID=0 PID=1311073 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-user-sau-main-dev-coordinator
[2026-01-19 13:27:37 UTC] USER=www-data EUID=0 PID=1311082 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/user-sau-main-dev/coordinator --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.u11y6p
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /data/postgresql/17/user-sau-main-dev/coordinator ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/user-sau-main-dev/coordinator -l logfile start

[0;32m[OK][0m   initdb complete
[2026-01-19 13:27:38 UTC] USER=www-data EUID=0 PID=1311137 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.u11y6p
[0;34m[INFO][0m Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[0;32m[OK][0m   postgresql.conf updated successfully
[0;34m[INFO][0m Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-19 13:27:38 UTC] USER=www-data EUID=0 PID=1311188 ACTION=fsop ARGS=cp /tmp/tmp.F4LfP35an6 /data/postgresql/17/user-sau-main-dev/coordinator/pg_hba.conf
[2026-01-19 13:27:38 UTC] USER=www-data EUID=0 PID=1311209 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev/coordinator/pg_hba.conf
[2026-01-19 13:27:38 UTC] USER=www-data EUID=0 PID=1311230 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/user-sau-main-dev/coordinator/pg_hba.conf
[0;32m[OK][0m   pg_hba.conf updated
[0;34m[INFO][0m Creating systemd unit: /etc/systemd/system/postgresql@user-sau-main-dev-coordinator.service
[2026-01-19 13:27:38 UTC] USER=www-data EUID=0 PID=1311255 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.dPggLM /etc/systemd/system/postgresql@user-sau-main-dev-coordinator.service
[2026-01-19 13:27:38 UTC] USER=www-data EUID=0 PID=1311276 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@user-sau-main-dev-coordinator.service
[0;32m[OK][0m   systemd unit written
[2026-01-19 13:27:38 UTC] USER=www-data EUID=0 PID=1311298 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-19 13:27:38 UTC] USER=www-data EUID=0 PID=1311319 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-19 13:27:38 UTC] USER=www-data EUID=0 PID=1311340 ACTION=passthru ARGS=systemctl daemon-reload
[0;34m[INFO][0m Starting PostgreSQL instance...
[2026-01-19 13:27:40 UTC] USER=www-data EUID=0 PID=1311486 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-coordinator.service
[0;34m[INFO][0m Waiting for ACTIVE (systemd)…
[2026-01-19 13:27:40 UTC] USER=www-data EUID=0 PID=1311571 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[0;32m[OK][0m   Service ACTIVE
[0;34m[INFO][0m Waiting for port 5432 bind…
[0;32m[OK][0m   Port bound
[0;34m[INFO][0m Waiting pg_isready (socket)…
[0;32m[OK][0m   Readiness via socket OK
[0;34m[INFO][0m Waiting pg_isready (TCP db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432)…
[0;32m[OK][0m   Startup sequence complete
[0;34m[INFO][0m Validating core security GUCs (via local socket)…
[0;32m[OK][0m   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[0;34m[INFO][0m Provisioning application database and Debezium role (if not exists)...
[0;34m[INFO][0m Checking if database fastorder_user_sau_main_dev_db exists...
[0;34m[INFO][0m DB check result: exit_code=0, output='[2026-01-19 13:27:41 UTC] USER=www-data EUID=0 PID=1312209 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_user_sau_main_dev_db''
[0;34m[INFO][0m Creating database fastorder_user_sau_main_dev_db...
[2026-01-19 13:27:41 UTC] USER=www-data EUID=0 PID=1312307 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_user_sau_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[0;32m[OK][0m   Database fastorder_user_sau_main_dev_db created
[0;34m[INFO][0m Checking if role debezium_user exists...
[0;34m[INFO][0m Role check result: exit_code=0, output='[2026-01-19 13:27:41 UTC] USER=www-data EUID=0 PID=1312437 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[0;34m[INFO][0m Creating role debezium_user...
[2026-01-19 13:27:42 UTC] USER=www-data EUID=0 PID=1312521 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'caE56zlCk0AncXYr2dVHDvlO';
CREATE ROLE
[0;32m[OK][0m   Role debezium_user created
[2026-01-19 13:27:42 UTC] USER=www-data EUID=0 PID=1312561 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_user_sau_main_dev_db" TO debezium_user;
GRANT
[0;32m[OK][0m   Application DB (fastorder_user_sau_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[0;34m[INFO][0m Applying connection and memory optimizations...
[0;34m[INFO][0m Current settings: max_connections=100, work_mem=4MB
[0;34m[INFO][0m Target settings (coordinator): max_connections=150, work_mem=8MB
[2026-01-19 13:27:42 UTC] USER=www-data EUID=0 PID=1312642 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 150;
ALTER SYSTEM
[2026-01-19 13:27:42 UTC] USER=www-data EUID=0 PID=1312665 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-19 13:27:42 UTC] USER=www-data EUID=0 PID=1312688 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[0;32m[OK][0m   Settings applied to postgresql.auto.conf
[2026-01-19 13:27:42 UTC] USER=www-data EUID=0 PID=1312703 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/user-sau-main-dev/coordinator/standby.signal
[0;34m[INFO][0m Service recently started (2s ago) - restarting to apply max_connections...
[0;34m[INFO][0m Stopping service...
[2026-01-19 13:27:43 UTC] USER=www-data EUID=0 PID=1312725 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-coordinator.service
[0;34m[INFO][0m Waiting for port 5432 to be released...
[0;32m[OK][0m   Port 5432 released
[0;34m[INFO][0m Starting service...
[2026-01-19 13:27:46 UTC] USER=www-data EUID=0 PID=1312894 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-coordinator.service
[2026-01-19 13:27:52 UTC] USER=www-data EUID=0 PID=1313240 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[0;32m[OK][0m   ✅ Optimization complete: max_connections=150, work_mem=8MB
[0;34m[INFO][0m Setting postgres password via centralized script... for coordinator
[0;34m[INFO][0m Temporarily disabling synchronous_commit on coordinator for password setting...
[0;32m[OK][0m   Disabled synchronous_commit (was: on)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    user[0m
[0;34m  Zone:       sau[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: coordinator[0m

[0;34mAWS Secret: fastorder/db/user/sau/main/dev/postgresql/coordinator[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-user-sau-main-dev-coordinator[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mInitial setup: Using password from initdb[0m
[0;32m✓ PostgreSQL password already set during initdb[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/coordinator
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/user/sau/main/dev/postgresql/coordinator[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials coordinator[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
[0;34m[INFO][0m Restoring synchronous_commit on coordinator...
[0;32m[OK][0m   Restored synchronous_commit to: on
[0;32m[OK][0m   Password set and persisted
[0;34m[INFO][0m Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.231
[0;34m[INFO][0m Primary hostname: db-user-sau-main-dev-postgresql-coordinator.fastorder.com

[0;34m[INFO][0m Adding /etc/hosts entries for coordinator...
[0;34m[INFO][0m   1. db-user-sau-main-dev-postgresql.fastorder.com → 10.100.1.231 (primary/short)
[0;34m[INFO][0m   2. db-user-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.231 (compatibility)

[0;34m[INFO][0m   ✅ db-user-sau-main-dev-postgresql.fastorder.com already exists with correct IP
[0;34m[INFO][0m   ✅ db-user-sau-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;32m✅[0m   ✅ Network & DNS configuration complete
[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
  10.100.1.231    db-user-sau-main-dev-postgresql-coordinator.fastorder.com
  10.100.1.231    db-user-sau-main-dev-postgresql.fastorder.com


[0;32m[OK][0m   PostgreSQL 'user-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key \
        host=db-user-sau-main-dev-postgresql-coordinator port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[0;34m[INFO][0m Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        user-sau-main-dev-postgresql-coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.231
[INFO]   Port:              5432
[INFO]   FQDN:              db-user-sau-main-dev-postgresql-coordinator
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 83d5bc7d-3699-4f7e-98b2-72fdfea60e05
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[0;32m[OK][0m   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 📦 03 role...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-19 13:28:04 UTC] USER=www-data EUID=0 PID=1314120 ACTION=fsop ARGS=test -f /data/postgresql/17/user-sau-main-dev/coordinator/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    debezium_user
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   debezium_user
  Hostname:    db-user-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:28:05 UTC] USER=www-data EUID=0 PID=1314296 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-debezium_user
[2026-01-19 13:28:05 UTC] USER=www-data EUID=0 PID=1314306 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-01-19 13:28:05 UTC] USER=www-data EUID=0 PID=1314315 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-debezium_user/ra_root.key
[2026-01-19 13:28:05 UTC] USER=www-data EUID=0 PID=1314324 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-01-19 13:28:05 UTC] USER=www-data EUID=0 PID=1314333 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:28:06 UTC] USER=www-data EUID=0 PID=1314365 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:28:06 UTC] USER=www-data EUID=0 PID=1314374 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:28:06 UTC] USER=www-data EUID=0 PID=1314383 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-19 13:28:06 UTC] USER=www-data EUID=0 PID=1314392 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt
[2026-01-19 13:28:06 UTC] USER=www-data EUID=0 PID=1314401 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:28:06 UTC] USER=www-data EUID=0 PID=1314410 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:28:06 UTC] USER=www-data EUID=0 PID=1314419 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-19 13:28:06 UTC] USER=www-data EUID=0 PID=1314429 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-19 13:28:06 UTC] USER=www-data EUID=0 PID=1314439 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der
[2026-01-19 13:28:06 UTC] USER=www-data EUID=0 PID=1314466 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:28:07 UTC] USER=www-data EUID=0 PID=1314476 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:28:07 UTC] USER=www-data EUID=0 PID=1314485 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-19 13:28:07 UTC] USER=www-data EUID=0 PID=1314494 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-19 13:28:07 UTC] USER=www-data EUID=0 PID=1314503 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-19 13:28:07 UTC] USER=www-data EUID=0 PID=1314512 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der
[2026-01-19 13:28:07 UTC] USER=www-data EUID=0 PID=1314521 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:28:07 UTC] USER=www-data EUID=0 PID=1314540 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:07 UTC] USER=www-data EUID=0 PID=1314575 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:07 UTC] USER=www-data EUID=0 PID=1314584 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:07 UTC] USER=www-data EUID=0 PID=1314593 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:07 UTC] USER=www-data EUID=0 PID=1314602 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:07 UTC] USER=www-data EUID=0 PID=1314611 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-19 13:28:07 UTC] USER=www-data EUID=0 PID=1314620 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt
[2026-01-19 13:28:07 UTC] USER=www-data EUID=0 PID=1314629 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:28:07 UTC] USER=www-data EUID=0 PID=1314639 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:28:07 UTC] USER=www-data EUID=0 PID=1314648 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-19 13:28:07 UTC] USER=www-data EUID=0 PID=1314666 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_pk8.der
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314676 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314686 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314695 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314704 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314713 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314722 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314732 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314741 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314750 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314759 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314768 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314777 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314787 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_pk8.der
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314827 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314836 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314858 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314867 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314876 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt
[2026-01-19 13:28:08 UTC] USER=www-data EUID=0 PID=1314894 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1314912 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1314921 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_pk8.der
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1314931 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1314950 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1314968 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1314990 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1315004 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1315017 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1315026 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1315035 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1315045 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1315064 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1315074 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1315083 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1315092 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1315102 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1315120 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:28:09 UTC] USER=www-data EUID=0 PID=1315138 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: debezium_user
Node: coordinator
FQDN: db-user-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-coordinator.fastorder.com -U debezium_user -d postgres

✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    user[0m
[0;34m  Zone:       sau[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: coordinator[0m

[0;34mAWS Secret: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-user-sau-main-dev-coordinator[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mGenerating new secure password...[0m
[0;34mUser fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: fastorder_admin_gd[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials coordinator[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: coordinator/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-user-sau-main-dev-postgresql.fastorder.com
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    fastorder_admin_gd
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   fastorder_admin_gd
  Hostname:    db-user-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:28:18 UTC] USER=www-data EUID=0 PID=1315661 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-fastorder_admin_gd
[2026-01-19 13:28:18 UTC] USER=www-data EUID=0 PID=1315670 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-19 13:28:18 UTC] USER=www-data EUID=0 PID=1315679 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
[2026-01-19 13:28:18 UTC] USER=www-data EUID=0 PID=1315688 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-19 13:28:18 UTC] USER=www-data EUID=0 PID=1315697 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315736 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315745 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315757 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315767 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315777 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315786 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315795 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315804 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315813 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315822 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315831 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315840 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315849 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315859 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315868 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315877 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315886 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315895 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315905 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315914 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315923 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315966 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315975 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315984 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1315993 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1316011 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1316020 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1316029 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:28:19 UTC] USER=www-data EUID=0 PID=1316038 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316047 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316056 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316065 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316075 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316085 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316096 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316106 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316115 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316126 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316135 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316144 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316153 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316162 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316171 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316181 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316190 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316200 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316215 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316224 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316233 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316242 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316251 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316260 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316269 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316278 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316287 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316297 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316307 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316316 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316326 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316336 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316345 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316354 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316363 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316374 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316383 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316392 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316401 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316410 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316419 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316428 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316437 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-19 13:28:20 UTC] USER=www-data EUID=0 PID=1316447 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:28:21 UTC] USER=www-data EUID=0 PID=1316457 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:28:21 UTC] USER=www-data EUID=0 PID=1316466 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:28:21 UTC] USER=www-data EUID=0 PID=1316475 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:28:21 UTC] USER=www-data EUID=0 PID=1316484 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:28:21 UTC] USER=www-data EUID=0 PID=1316493 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:28:21 UTC] USER=www-data EUID=0 PID=1316503 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:28:21 UTC] USER=www-data EUID=0 PID=1316512 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:28:21 UTC] USER=www-data EUID=0 PID=1316521 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:28:21 UTC] USER=www-data EUID=0 PID=1316530 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: fastorder_admin_gd
Node: coordinator
FQDN: db-user-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-coordinator.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-user-sau-main-dev-coordinator:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_user_sau_main_dev_db already exists, skipping creation
[2026-01-19 13:28:21 UTC] USER=www-data EUID=0 PID=1316595 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /data/postgresql/17/user-sau-main-dev/coordinator/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-19 13:28:21 UTC] USER=www-data EUID=0 PID=1316632 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-coordinator.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

[0;34m=== Pre-flight Checks ===[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;32m✓[0m AWS Secrets Manager accessible

[0;34m=== Retrieving Credentials from AWS ===[0m
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials loaded for coordinator/fastorder_admin_gd: fastorder_admin_gd@db-user-sau-main-dev-postgresql.fastorder.com:5432/fastorder_user_sau_main_dev_db
[0;32m✓[0m Credentials retrieved: fastorder_admin_gd@db-user-sau-main-dev-postgresql.fastorder.com:5432/fastorder_user_sau_main_dev_db
[0;34m╔════════════════════════════════════════════╗[0m
[0;34m║  PostgreSQL Test Suite (AWS Secrets MGR)  ║[0m
[0;34m╚════════════════════════════════════════════╝[0m

[0;34m=== PostgreSQL Authentication Test ===[0m
[0;31m✗[0m PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-user-sau-main-dev-postgresql.fastorder.com" (10.100.1.231), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-19 13:28:26 UTC] USER=www-data EUID=0 PID=1316856 ACTION=fsop ARGS=test -f /data/postgresql/17/user-sau-main-dev/coordinator/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : user-sau-main-dev
  IDENTIFIER  : coordinator
  PG HOST     : db-user-sau-main-dev-postgresql.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_user_sau_main_dev_db
  SCHEMA      : user
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
  DNS → 10.100.1.231
  CA         : /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    user[0m
[0;34m  Zone:       sau[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: coordinator[0m

[0;34mAWS Secret: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-user-sau-main-dev-coordinator[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mGenerating new secure password...[0m
[0;34mUser debezium_user does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: debezium_user[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials coordinator[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: coordinator/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-01-19 13:28:33 UTC] USER=www-data EUID=0 PID=1317356 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_user_sau_main_dev_db already exists
[2026-01-19 13:28:34 UTC] USER=www-data EUID=0 PID=1317401 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d fastorder_user_sau_main_dev_db --no-psqlrc
ERROR:  syntax error at or near "user"
LINE 1: CREATE SCHEMA IF NOT EXISTS user;
                                    ^
GRANT
ERROR:  syntax error at or near "user"
LINE 1: GRANT USAGE ON SCHEMA user TO debezium_user;
                              ^
ERROR:  syntax error at or near "user"
LINE 1: GRANT SELECT ON ALL TABLES IN SCHEMA user TO debezium_user;
                                             ^
ERROR:  syntax error at or near "user"
LINE 1: GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA user TO debez...
                                                       ^
ERROR:  syntax error at or near "user"
LINE 1: ALTER DEFAULT PRIVILEGES IN SCHEMA user GRANT SELECT ON TABL...
                                           ^
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/user-sau-main-dev/coordinator/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (user) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=user
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/user/run.sh
🔍 DEBUG_CHECKPOINT_03: No specific folder for user, using default
[DEBUG] Tracking substep start: steps/01-install/steps/default (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 🔸 Service: user (using default contracts schema)
🔍 DEBUG_CHECKPOINT_04: Executing default: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/run.sh
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting default contracts schema provisioning for SERVICE=user
[INFO] Environment: user-sau-main-dev
[INFO] Schema: user (contracts tables)
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG: Looking for contracts steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/default/contracts/steps
[INFO] 📁 Running contracts schema setup for: user
[INFO] 📁 Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/default/contracts/steps

[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing user schema (contracts tables)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Schema:      user
  Identifier:  coordinator
  Database:    fastorder_user_sau_main_dev_db
  Host:        db-user-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_user_sau_main_dev_db
ℹ️  Database fastorder_user_sau_main_dev_db already exists
✅ Connected to database: fastorder_user_sau_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
CREATE EXTENSION
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating user schema...
CREATE SCHEMA
✅ Schema created
🔧 Creating contracts tables in user schema...
   Creating "user".contract_key...
CREATE TABLE
   Creating "user".contract_type...
CREATE TABLE
   Creating "user".contracts...
CREATE TABLE
   Adding columns to "user".contracts (safe migration)...
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
UPDATE 0
UPDATE 0
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
   Creating "user".contract_vars...
CREATE TABLE
   Creating "user".contract_datetime...
CREATE TABLE
   Creating "user".contract_decimal...
CREATE TABLE
   Creating "user".contract_float...
CREATE TABLE
   Creating "user".contract_int...
CREATE TABLE
   Creating "user".contract_json...
CREATE TABLE
   Creating "user".contract_terms...
CREATE TABLE
   Creating "user".contract_term_contracts...
CREATE TABLE
   Creating "user".contract_term_datetime...
CREATE TABLE
   Creating "user".contract_term_decimal...
CREATE TABLE
   Creating "user".contract_term_float...
CREATE TABLE
   Creating "user".contract_term_int...
CREATE TABLE
   Creating "user".contract_term_items...
CREATE TABLE
   Creating "user".contract_term_json...
CREATE TABLE
   Creating "user".contract_term_vars...
CREATE TABLE
   Creating "user".user_id_uuid_mapping...
CREATE TABLE
✅ All 19 tables created
🔧 Creating indexes...
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
✅ All indexes created
🔧 Creating foreign keys...
DO
DO
✅ Foreign keys created
🔧 Configuring Citus distribution...
   Creating reference table: contract_key
 create_reference_table 
------------------------
 
(1 row)

   Creating reference table: contract_type
 create_reference_table 
------------------------
 
(1 row)

   Creating distributed table: contracts
   Creating distributed table: contract_vars
   Creating distributed table: contract_datetime
   Creating distributed table: contract_decimal
   Creating distributed table: contract_float
   Creating distributed table: contract_int
   Creating distributed table: contract_json
   Creating distributed table: contract_terms
   Creating distributed table: contract_term_contracts
   Creating distributed table: contract_term_datetime
   Creating distributed table: contract_term_decimal
   Creating distributed table: contract_term_float
   Creating distributed table: contract_term_int
   Creating distributed table: contract_term_items
   Creating distributed table: contract_term_json
 create_distributed_table 
--------------------------
 
(1 row)

   Creating distributed table: contract_term_vars
 create_distributed_table 
--------------------------
 
(1 row)

✅ Citus distribution configured
🎉 Schema initialization complete for user in fastorder_user_sau_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

==========================================
✅ user schema initialization complete!
   Tables: 19
   Indexes: 54
==========================================

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Default contracts schema setup complete for: user
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[0;32m✓[0m ✅ Coordinator setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up 1 worker(s) (Citus data nodes)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up worker: worker-01
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-19 13:28:50 UTC] USER=unknown EUID=33 PID=1318250 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-19 13:28:50 UTC] USER=unknown EUID=33 PID=1318257 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-19 13:28:50 UTC] USER=unknown EUID=33 PID=1318270 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-19 13:28:50 UTC] USER=unknown EUID=33 PID=1318280 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-19 13:28:50 UTC] USER=unknown EUID=33 PID=1318287 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-19 13:28:50 UTC] USER=unknown EUID=33 PID=1318294 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for user in sau-dev...
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4 JOB_UUID=01c92b49-cb3d-406f-92b1-bfc5cb7c45c4

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.232
[0;34m[INFO][0m Primary hostname: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

[0;34m[INFO][0m Adding /etc/hosts entry for worker-01...
[0;34m[INFO][0m   db-user-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.232

[0;34m[INFO][0m   ✅ db-user-sau-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;32m✅[0m   ✅ Network & DNS configuration complete
[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
  10.100.1.232    db-user-sau-main-dev-postgresql-worker-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-user-sau-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      user-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-19 13:28:53 UTC] USER=www-data EUID=0 PID=1318441 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01 and /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:28:53 UTC] USER=www-data EUID=0 PID=1318450 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-19 13:28:53 UTC] USER=www-data EUID=0 PID=1318460 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1318405
[2026-01-19 13:28:53 UTC] USER=www-data EUID=0 PID=1318487 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1318405/ra_root.crt
[2026-01-19 13:28:53 UTC] USER=www-data EUID=0 PID=1318496 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1318405/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[2026-01-19 13:28:54 UTC] USER=www-data EUID=0 PID=1318540 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1318405/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-19 13:28:54 UTC] USER=www-data EUID=0 PID=1318560 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-19 13:28:54 UTC] USER=www-data EUID=0 PID=1318576 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1318405/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:28:54 UTC] USER=www-data EUID=0 PID=1318598 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:28:54 UTC] USER=www-data EUID=0 PID=1318607 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-19 13:28:54 UTC] USER=www-data EUID=0 PID=1318618 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-19 13:28:54 UTC] USER=www-data EUID=0 PID=1318627 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-19 13:28:54 UTC] USER=www-data EUID=0 PID=1318636 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-19 13:28:54 UTC] USER=www-data EUID=0 PID=1318645 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-19 13:28:54 UTC] USER=www-data EUID=0 PID=1318654 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-user-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:user-sau-main-dev.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: user-sau-main-dev
Node:        worker-01
Primary CN:  db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-user-sau-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318721 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318742 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318751 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318760 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318786 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318795 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318804 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318813 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318822 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318831 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318840 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318849 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318858 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318867 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318878 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318887 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318896 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318923 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:28:55 UTC] USER=www-data EUID=0 PID=1318932 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1318942 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1318968 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1318978 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1318987 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319009 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319018 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319027 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319036 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319045 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319054 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319063 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319072 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319082 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319092 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319101 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319112 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319121 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319130 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319139 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319148 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319157 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319166 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319176 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319187 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319199 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319209 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319219 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:56 UTC] USER=www-data EUID=0 PID=1319228 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319246 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319264 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319273 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319283 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319292 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319301 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319310 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319320 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319330 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319349 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319359 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319372 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319382 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319391 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319400 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319409 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319418 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319427 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319436 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319445 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319455 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319467 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319476 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319485 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319494 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319503 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319512 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319521 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319530 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:28:57 UTC] USER=www-data EUID=0 PID=1319539 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: postgres
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319608 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319620 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319629 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319638 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319647 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319749 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319769 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319778 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319788 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319797 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319806 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319815 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319824 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319833 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319842 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319853 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319878 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:28:58 UTC] USER=www-data EUID=0 PID=1319888 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1319898 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1319907 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1319916 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1319925 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1319934 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1319943 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1319952 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1319992 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320013 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320027 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320045 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320055 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320065 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320074 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320084 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320093 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320102 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320111 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320120 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320130 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320140 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320149 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320158 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320167 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320176 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320185 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320195 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320204 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320213 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320223 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320236 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320249 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:28:59 UTC] USER=www-data EUID=0 PID=1320259 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320270 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320283 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320302 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320313 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320322 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320331 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320340 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320349 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320369 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320378 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320388 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320399 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320424 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320437 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320446 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320455 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320464 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320473 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320482 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320491 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320501 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320519 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320528 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320548 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320557 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320566 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320584 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320593 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320602 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:29:00 UTC] USER=www-data EUID=0 PID=1320612 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:29:01 UTC] USER=www-data EUID=0 PID=1320621 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: postgres
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 📦 02 setup pg instance...
[0;34m[DEADLOCK-PREVENTION][0m Deadlock prevention library loaded
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-postgresql environment: db-user-sau-main-dev-postgresql-worker-01.fastorder.com (10.100.1.232)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.232
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m Data dir:   /data/postgresql/17/user-sau-main-dev/worker-01
[0;34m[INFO][0m Port:       5432
[0;34m[INFO][0m Hostname:   db-user-sau-main-dev-postgresql-worker-01
[2026-01-19 13:29:02 UTC] USER=www-data EUID=0 PID=1320769 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:29:02 UTC] USER=www-data EUID=0 PID=1320790 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:29:02 UTC] USER=www-data EUID=0 PID=1320811 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:29:02 UTC] USER=www-data EUID=0 PID=1320833 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[1;33m[WARN][0m Server certificate not found at /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[0;34m[INFO][0m Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-user-sau-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      user-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-19 13:29:03 UTC] USER=www-data EUID=0 PID=1320912 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01 and /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:29:03 UTC] USER=www-data EUID=0 PID=1320921 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-19 13:29:03 UTC] USER=www-data EUID=0 PID=1320931 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1320840
[2026-01-19 13:29:03 UTC] USER=www-data EUID=0 PID=1320940 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1320840/ra_root.crt
[2026-01-19 13:29:03 UTC] USER=www-data EUID=0 PID=1320949 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1320840/ra_root.key
[2026-01-19 13:29:03 UTC] USER=www-data EUID=0 PID=1320958 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1320840/ra_root.crt
[2026-01-19 13:29:03 UTC] USER=www-data EUID=0 PID=1320967 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1320840/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[2026-01-19 13:29:04 UTC] USER=www-data EUID=0 PID=1321021 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1320840/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-19 13:29:04 UTC] USER=www-data EUID=0 PID=1321030 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1320840/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-19 13:29:04 UTC] USER=www-data EUID=0 PID=1321039 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-19 13:29:04 UTC] USER=www-data EUID=0 PID=1321048 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1320840/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:04 UTC] USER=www-data EUID=0 PID=1321057 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:04 UTC] USER=www-data EUID=0 PID=1321066 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:04 UTC] USER=www-data EUID=0 PID=1321075 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-19 13:29:04 UTC] USER=www-data EUID=0 PID=1321088 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-19 13:29:04 UTC] USER=www-data EUID=0 PID=1321097 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-19 13:29:04 UTC] USER=www-data EUID=0 PID=1321106 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-19 13:29:04 UTC] USER=www-data EUID=0 PID=1321115 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-19 13:29:04 UTC] USER=www-data EUID=0 PID=1321124 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:29:04 UTC] USER=www-data EUID=0 PID=1321133 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-user-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:user-sau-main-dev.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: user-sau-main-dev
Node:        worker-01
Primary CN:  db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-user-sau-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-19 13:29:04 UTC] USER=www-data EUID=0 PID=1321162 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-19 13:29:04 UTC] USER=www-data EUID=0 PID=1321171 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-19 13:29:04 UTC] USER=www-data EUID=0 PID=1321180 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[0;32m[OK][0m   mTLS certificates OK (server cert + client certs verified) and keys secured
[0;34m[INFO][0m Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-19 13:29:04 UTC] USER=www-data EUID=0 PID=1321201 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-worker-01.service
[2026-01-19 13:29:05 UTC] USER=www-data EUID=0 PID=1321239 ACTION=passthru ARGS=systemctl stop postgresql
[1;33m[WARN][0m Cleaning stale socket directory /var/run/postgresql-user-sau-main-dev-worker-01
[2026-01-19 13:29:05 UTC] USER=www-data EUID=0 PID=1321270 ACTION=fsop ARGS=rm -rf /var/run/postgresql-user-sau-main-dev-worker-01
[0;32m[OK][0m   No conflicting Postgres left on port 5432
[0;32m[OK][0m   Using postgres password from vault provider
[2026-01-19 13:29:06 UTC] USER=www-data EUID=0 PID=1321338 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.WfAcMn
[2026-01-19 13:29:06 UTC] USER=www-data EUID=0 PID=1321363 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.WfAcMn
[2026-01-19 13:29:06 UTC] USER=www-data EUID=0 PID=1321387 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/user-sau-main-dev
[2026-01-19 13:29:06 UTC] USER=www-data EUID=0 PID=1321409 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev
[2026-01-19 13:29:06 UTC] USER=www-data EUID=0 PID=1321433 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/user-sau-main-dev
[0;34m[INFO][0m Initializing cluster in /data/postgresql/17/user-sau-main-dev/worker-01 (SCRAM; pwfile)
[1;33m[WARN][0m Removing existing data directory: /data/postgresql/17/user-sau-main-dev/worker-01
[2026-01-19 13:29:07 UTC] USER=www-data EUID=0 PID=1321454 ACTION=fsop ARGS=rm -rf /data/postgresql/17/user-sau-main-dev/worker-01
[2026-01-19 13:29:07 UTC] USER=www-data EUID=0 PID=1321476 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/user-sau-main-dev/worker-01
[2026-01-19 13:29:07 UTC] USER=www-data EUID=0 PID=1321497 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev/worker-01
[2026-01-19 13:29:07 UTC] USER=www-data EUID=0 PID=1321542 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-user-sau-main-dev-worker-01
[2026-01-19 13:29:07 UTC] USER=www-data EUID=0 PID=1321563 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-user-sau-main-dev-worker-01
[2026-01-19 13:29:07 UTC] USER=www-data EUID=0 PID=1321584 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-user-sau-main-dev-worker-01
[2026-01-19 13:29:07 UTC] USER=www-data EUID=0 PID=1321593 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/user-sau-main-dev/worker-01 --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.WfAcMn
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /data/postgresql/17/user-sau-main-dev/worker-01 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/user-sau-main-dev/worker-01 -l logfile start

[0;32m[OK][0m   initdb complete
[2026-01-19 13:29:08 UTC] USER=www-data EUID=0 PID=1321641 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.WfAcMn
[0;34m[INFO][0m Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[0;32m[OK][0m   postgresql.conf updated successfully
[0;34m[INFO][0m Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-19 13:29:08 UTC] USER=www-data EUID=0 PID=1321701 ACTION=fsop ARGS=cp /tmp/tmp.beySnGhtw9 /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[2026-01-19 13:29:08 UTC] USER=www-data EUID=0 PID=1321722 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[2026-01-19 13:29:08 UTC] USER=www-data EUID=0 PID=1321743 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[0;32m[OK][0m   pg_hba.conf updated
[0;34m[INFO][0m Creating systemd unit: /etc/systemd/system/postgresql@user-sau-main-dev-worker-01.service
[2026-01-19 13:29:08 UTC] USER=www-data EUID=0 PID=1321768 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.txjVY7 /etc/systemd/system/postgresql@user-sau-main-dev-worker-01.service
[2026-01-19 13:29:08 UTC] USER=www-data EUID=0 PID=1321791 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@user-sau-main-dev-worker-01.service
[0;32m[OK][0m   systemd unit written
[2026-01-19 13:29:08 UTC] USER=www-data EUID=0 PID=1321812 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-19 13:29:08 UTC] USER=www-data EUID=0 PID=1321833 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-19 13:29:08 UTC] USER=www-data EUID=0 PID=1321854 ACTION=passthru ARGS=systemctl daemon-reload
[0;34m[INFO][0m Starting PostgreSQL instance...
[2026-01-19 13:29:09 UTC] USER=www-data EUID=0 PID=1321981 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-worker-01.service
[0;34m[INFO][0m Waiting for ACTIVE (systemd)…
[2026-01-19 13:29:10 UTC] USER=www-data EUID=0 PID=1322037 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-worker-01.service
[0;32m[OK][0m   Service ACTIVE
[0;34m[INFO][0m Waiting for port 5432 bind…
[0;32m[OK][0m   Port bound
[0;34m[INFO][0m Waiting pg_isready (socket)…
[0;32m[OK][0m   Readiness via socket OK
[0;34m[INFO][0m Waiting pg_isready (TCP db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432)…
[0;32m[OK][0m   Startup sequence complete
[0;34m[INFO][0m Validating core security GUCs (via local socket)…
[0;32m[OK][0m   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[0;34m[INFO][0m Provisioning application database and Debezium role (if not exists)...
[0;34m[INFO][0m Checking if database fastorder_user_sau_main_dev_db exists...
[0;34m[INFO][0m DB check result: exit_code=0, output='[2026-01-19 13:29:11 UTC] USER=www-data EUID=0 PID=1322242 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_user_sau_main_dev_db''
[0;34m[INFO][0m Creating database fastorder_user_sau_main_dev_db...
[2026-01-19 13:29:11 UTC] USER=www-data EUID=0 PID=1322265 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_user_sau_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[0;32m[OK][0m   Database fastorder_user_sau_main_dev_db created
[0;34m[INFO][0m Checking if role debezium_user exists...
[0;34m[INFO][0m Role check result: exit_code=0, output='[2026-01-19 13:29:11 UTC] USER=www-data EUID=0 PID=1322290 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[0;34m[INFO][0m Creating role debezium_user...
[2026-01-19 13:29:11 UTC] USER=www-data EUID=0 PID=1322317 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'h6VJIW+1+bndfNqpjcu2OXs3';
CREATE ROLE
[0;32m[OK][0m   Role debezium_user created
[2026-01-19 13:29:12 UTC] USER=www-data EUID=0 PID=1322340 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_user_sau_main_dev_db" TO debezium_user;
GRANT
[0;32m[OK][0m   Application DB (fastorder_user_sau_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[0;34m[INFO][0m Applying connection and memory optimizations...
[0;34m[INFO][0m Current settings: max_connections=100, work_mem=4MB
[0;34m[INFO][0m Target settings (worker): max_connections=100, work_mem=8MB
[2026-01-19 13:29:12 UTC] USER=www-data EUID=0 PID=1322453 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 100;
ALTER SYSTEM
[2026-01-19 13:29:12 UTC] USER=www-data EUID=0 PID=1322489 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-19 13:29:12 UTC] USER=www-data EUID=0 PID=1322523 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[0;32m[OK][0m   Settings applied to postgresql.auto.conf
[2026-01-19 13:29:12 UTC] USER=www-data EUID=0 PID=1322560 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/user-sau-main-dev/worker-01/standby.signal
[0;34m[INFO][0m Service recently started (2s ago) - restarting to apply max_connections...
[0;34m[INFO][0m Stopping service...
[2026-01-19 13:29:13 UTC] USER=www-data EUID=0 PID=1322599 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-worker-01.service
[0;34m[INFO][0m Waiting for port 5432 to be released...
[0;32m[OK][0m   Port 5432 released
[0;34m[INFO][0m Starting service...
[2026-01-19 13:29:16 UTC] USER=www-data EUID=0 PID=1322749 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-worker-01.service
[2026-01-19 13:29:22 UTC] USER=www-data EUID=0 PID=1322858 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-worker-01.service
[0;32m[OK][0m   ✅ Optimization complete: max_connections=100, work_mem=8MB
[0;32m[OK][0m   Synchronous replication already configured (synchronous_commit: on)
[0;34m[INFO][0m Setting postgres password via centralized script... for worker-01
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    user[0m
[0;34m  Zone:       sau[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: worker-01[0m

[0;34mAWS Secret: fastorder/db/user/sau/main/dev/postgresql/worker-01[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-user-sau-main-dev-worker-01[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mInitial setup: Using password from initdb[0m
[0;32m✓ PostgreSQL password already set during initdb[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/worker-01
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/user/sau/main/dev/postgresql/worker-01[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials worker-01[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
[0;32m[OK][0m   Password set and persisted
[0;34m[INFO][0m Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.232
[0;34m[INFO][0m Primary hostname: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

[0;34m[INFO][0m Adding /etc/hosts entry for worker-01...
[0;34m[INFO][0m   db-user-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.232

[0;34m[INFO][0m   ✅ db-user-sau-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;32m✅[0m   ✅ Network & DNS configuration complete
[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
  10.100.1.232    db-user-sau-main-dev-postgresql-worker-01.fastorder.com


[0;32m[OK][0m   PostgreSQL 'user-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key \
        host=db-user-sau-main-dev-postgresql-worker-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[0;34m[INFO][0m Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        user-sau-main-dev-postgresql-worker-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.232
[INFO]   Port:              5432
[INFO]   FQDN:              db-user-sau-main-dev-postgresql-worker-01
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 87ccba48-d8e0-43e4-97b8-d87917a5d35c
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[0;32m[OK][0m   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 📦 03 role...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    debezium_user
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   debezium_user
  Hostname:    db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:29:33 UTC] USER=www-data EUID=0 PID=1323575 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-debezium_user
[2026-01-19 13:29:33 UTC] USER=www-data EUID=0 PID=1323584 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323593 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-debezium_user/ra_root.key
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323602 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.crt
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323627 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323636 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323654 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323663 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323682 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323691 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323700 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_pk8.der
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323709 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323728 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323737 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323746 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323755 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323775 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323810 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323828 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323839 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323849 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323862 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-19 13:29:34 UTC] USER=www-data EUID=0 PID=1323872 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1323881 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1323890 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1323907 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1323920 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1323929 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_pk8.der
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1323939 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1323960 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1323980 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1323989 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1323998 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324007 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324016 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324025 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324034 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324043 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324052 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_pk8.der
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324062 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324072 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324082 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324091 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324100 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324118 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324127 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324155 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324164 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324173 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_pk8.der
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324183 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324193 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324202 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324211 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:35 UTC] USER=www-data EUID=0 PID=1324229 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324238 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324258 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324267 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324276 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324286 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324295 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324304 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_pk8.der
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324317 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324327 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324336 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
   ✅ Symlinked ca.pem
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324354 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324363 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324372 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324381 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324390 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324399 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: debezium_user
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U debezium_user -d postgres

🔐 Generating replicator client certificate for worker-01...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324443 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-19 13:29:36 UTC] USER=www-data EUID=0 PID=1324479 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324494 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324503 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324521 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324530 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324539 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324548 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324557 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324566 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324575 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324584 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324593 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324602 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324621 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324648 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324657 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324666 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324678 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324722 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324733 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324744 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324755 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:29:37 UTC] USER=www-data EUID=0 PID=1324764 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324782 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324792 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324801 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324810 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324820 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324837 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324847 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324856 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324865 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324874 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324883 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324892 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324901 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324910 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324919 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324928 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324937 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324946 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324956 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324966 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324975 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1324989 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325003 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325012 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325021 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325030 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325039 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325048 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325057 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325066 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325075 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325085 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325095 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325104 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325113 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325122 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325136 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325148 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325157 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325166 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325176 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:29:38 UTC] USER=www-data EUID=0 PID=1325186 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:29:39 UTC] USER=www-data EUID=0 PID=1325195 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:29:39 UTC] USER=www-data EUID=0 PID=1325204 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:29:39 UTC] USER=www-data EUID=0 PID=1325214 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:29:39 UTC] USER=www-data EUID=0 PID=1325224 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:29:39 UTC] USER=www-data EUID=0 PID=1325233 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:29:39 UTC] USER=www-data EUID=0 PID=1325242 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:29:39 UTC] USER=www-data EUID=0 PID=1325251 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:29:39 UTC] USER=www-data EUID=0 PID=1325260 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:29:39 UTC] USER=www-data EUID=0 PID=1325269 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:29:39 UTC] USER=www-data EUID=0 PID=1325279 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:29:39 UTC] USER=www-data EUID=0 PID=1325288 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:29:39 UTC] USER=www-data EUID=0 PID=1325297 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

✅ Replicator certificate generated for worker-01
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    user[0m
[0;34m  Zone:       sau[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: worker-01[0m

[0;34mAWS Secret: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-user-sau-main-dev-worker-01[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mGenerating new secure password...[0m
[0;34mUser fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: fastorder_admin_gd[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials worker-01[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: worker-01/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    fastorder_admin_gd
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   fastorder_admin_gd
  Hostname:    db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:29:46 UTC] USER=www-data EUID=0 PID=1325781 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-fastorder_admin_gd
[2026-01-19 13:29:46 UTC] USER=www-data EUID=0 PID=1325790 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-01-19 13:29:46 UTC] USER=www-data EUID=0 PID=1325799 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
[2026-01-19 13:29:46 UTC] USER=www-data EUID=0 PID=1325808 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-01-19 13:29:46 UTC] USER=www-data EUID=0 PID=1325817 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:29:46 UTC] USER=www-data EUID=0 PID=1325832 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:29:46 UTC] USER=www-data EUID=0 PID=1325841 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:29:46 UTC] USER=www-data EUID=0 PID=1325850 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-19 13:29:46 UTC] USER=www-data EUID=0 PID=1325859 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1325868 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1325877 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1325886 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1325895 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1325904 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1325913 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1325922 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1325931 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1325940 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1325949 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1325958 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1325967 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1325976 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1325986 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1325998 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326009 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326022 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326048 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326057 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326066 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326076 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326086 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326095 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326104 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326113 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326122 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326131 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326140 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326149 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326159 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326169 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326178 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326189 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326198 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326207 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326216 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326225 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326234 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326243 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326252 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326261 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-19 13:29:47 UTC] USER=www-data EUID=0 PID=1326271 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326281 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326291 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326300 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326309 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326319 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326328 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326337 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326348 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326357 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326366 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326375 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326384 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326393 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326403 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326413 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326422 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326431 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326441 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326450 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326459 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326468 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326478 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326487 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326496 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326510 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326523 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326533 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326543 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326552 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326561 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326570 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326579 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326588 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326597 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326606 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:29:48 UTC] USER=www-data EUID=0 PID=1326615 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: fastorder_admin_gd
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-user-sau-main-dev-worker-01:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_user_sau_main_dev_db already exists, skipping creation
[2026-01-19 13:29:49 UTC] USER=www-data EUID=0 PID=1326684 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-19 13:29:49 UTC] USER=www-data EUID=0 PID=1326721 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-worker-01.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

[0;34m=== Pre-flight Checks ===[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;32m✓[0m AWS Secrets Manager accessible

[0;34m=== Retrieving Credentials from AWS ===[0m
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials loaded for worker-01/fastorder_admin_gd: fastorder_admin_gd@db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_user_sau_main_dev_db
[0;32m✓[0m Credentials retrieved: fastorder_admin_gd@db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_user_sau_main_dev_db
[0;34m╔════════════════════════════════════════════╗[0m
[0;34m║  PostgreSQL Test Suite (AWS Secrets MGR)  ║[0m
[0;34m╚════════════════════════════════════════════╝[0m

[0;34m=== PostgreSQL Authentication Test ===[0m
[0;31m✗[0m PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-user-sau-main-dev-postgresql-worker-01.fastorder.com" (10.100.1.232), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
── fast setup ─────────────────────────────────────────────
  NAME        : user-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_user_sau_main_dev_db
  SCHEMA      : user
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
  DNS → 10.100.1.232
  CA         : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║   PostgreSQL Password Rotation via AWS Secrets Manager    ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m

[0;34mEnvironment Configuration:[0m
[0;34m  Service:    user[0m
[0;34m  Zone:       sau[0m
[0;34m  Environment: dev[0m
[0;34m  Identifier: worker-01[0m

[0;34mAWS Secret: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user[0m

[0;34mConnection Info:[0m
[0;34m  Socket Dir: /var/run/postgresql-user-sau-main-dev-worker-01[0m
[0;34m  Port:       5432[0m

[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m

[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m  ✓ Zero-downtime (dual-password window)[0m
[0;34m  ✓ Automatic rollback on failure[0m
[0;34m  ✓ CloudTrail audit log[0m
[0;34m  ✓ CloudWatch metrics[0m
[0;34m  ✓ No secret exposure in scripts[0m

[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m

[0;34mGenerating new secure password...[0m
[0;34mUser debezium_user does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: debezium_user[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user[0m
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user
[0;32m✓ Password stored in AWS Secrets Manager[0m

[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m

[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m

[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║              Password Rotation Complete!                   ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m

[0;34mSecret: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m

[0;34mTo retrieve credentials:[0m
[0;34m  # Using Bash library[0m
[0;34m  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m  get_pg_credentials worker-01[0m

[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m

[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: worker-01/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-01-19 13:30:02 UTC] USER=www-data EUID=0 PID=1328436 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_user_sau_main_dev_db already exists
[2026-01-19 13:30:02 UTC] USER=www-data EUID=0 PID=1328593 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d fastorder_user_sau_main_dev_db --no-psqlrc
ERROR:  syntax error at or near "user"
LINE 1: CREATE SCHEMA IF NOT EXISTS user;
                                    ^
GRANT
ERROR:  syntax error at or near "user"
LINE 1: GRANT USAGE ON SCHEMA user TO debezium_user;
                              ^
ERROR:  syntax error at or near "user"
LINE 1: GRANT SELECT ON ALL TABLES IN SCHEMA user TO debezium_user;
                                             ^
ERROR:  syntax error at or near "user"
LINE 1: GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA user TO debez...
                                                       ^
ERROR:  syntax error at or near "user"
LINE 1: ALTER DEFAULT PRIVILEGES IN SCHEMA user GRANT SELECT ON TABL...
                                           ^
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
🔐 Creating replicator role for worker-01...
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : user-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SSL DIR     : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
  DNS → 10.100.1.232
  CA         : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Creating role: replicator with password
SET
CREATE ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/user/sau/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🎉 Done.
✅ Replicator role created for worker-01

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (user) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=user
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/user/run.sh
🔍 DEBUG_CHECKPOINT_03: No specific folder for user, using default
[DEBUG] Tracking substep start: steps/01-install/steps/default (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 🔸 Service: user (using default contracts schema)
🔍 DEBUG_CHECKPOINT_04: Executing default: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/run.sh
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting default contracts schema provisioning for SERVICE=user
[INFO] Environment: user-sau-main-dev
[INFO] Schema: user (contracts tables)
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Skipping Schema Setup on worker-01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

ℹ️  Schema setup only runs on coordinator
ℹ️  This is a worker-01 node - schemas replicate automatically

✅ Nothing to do on this node

[0;32m✓[0m ✅ Worker worker-01 setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up standby replicas (1 per worker)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up standby: worker-01-standby-01 (replica of worker-01)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-19 13:30:11 UTC] USER=unknown EUID=33 PID=1332764 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-19 13:30:11 UTC] USER=unknown EUID=33 PID=1332811 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-19 13:30:11 UTC] USER=unknown EUID=33 PID=1332825 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-19 13:30:11 UTC] USER=unknown EUID=33 PID=1332851 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-19 13:30:11 UTC] USER=unknown EUID=33 PID=1332867 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-19 13:30:11 UTC] USER=unknown EUID=33 PID=1332902 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for user in sau-dev...
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4 JOB_UUID=01c92b49-cb3d-406f-92b1-bfc5cb7c45c4

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Identifier: worker-01-standby-01
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.233
[0;34m[INFO][0m Primary hostname: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

[0;34m[INFO][0m Adding /etc/hosts entry for worker-01-standby-01...
[0;34m[INFO][0m   db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.233

[0;34m[INFO][0m   ✅ db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com already exists with correct IP

[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;32m✅[0m   ✅ Network & DNS configuration complete
[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
  10.100.1.233    db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  user-sau-main-dev.fastorder.com
  Alt CN:      user-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-19 13:30:15 UTC] USER=www-data EUID=0 PID=1333740 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:15 UTC] USER=www-data EUID=0 PID=1333749 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-19 13:30:16 UTC] USER=www-data EUID=0 PID=1333763 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1333690
[2026-01-19 13:30:16 UTC] USER=www-data EUID=0 PID=1333773 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1333690/ra_root.crt
[2026-01-19 13:30:16 UTC] USER=www-data EUID=0 PID=1333782 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1333690/ra_root.key
[2026-01-19 13:30:16 UTC] USER=www-data EUID=0 PID=1333791 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1333690/ra_root.crt
[2026-01-19 13:30:16 UTC] USER=www-data EUID=0 PID=1333801 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1333690/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = user-sau-main-dev.fastorder.com
[2026-01-19 13:30:17 UTC] USER=www-data EUID=0 PID=1333877 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1333690/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
[2026-01-19 13:30:17 UTC] USER=www-data EUID=0 PID=1333895 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-19 13:30:17 UTC] USER=www-data EUID=0 PID=1333904 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1333690/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:17 UTC] USER=www-data EUID=0 PID=1333913 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:17 UTC] USER=www-data EUID=0 PID=1333922 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:17 UTC] USER=www-data EUID=0 PID=1333931 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-19 13:30:17 UTC] USER=www-data EUID=0 PID=1333942 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
[2026-01-19 13:30:17 UTC] USER=www-data EUID=0 PID=1333988 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: user-sau-main-dev
Node:        worker-01-standby-01
Primary CN:  user-sau-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=user-sau-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:30:18 UTC] USER=www-data EUID=0 PID=1334044 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-01-19 13:30:18 UTC] USER=www-data EUID=0 PID=1334053 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-19 13:30:18 UTC] USER=www-data EUID=0 PID=1334071 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-19 13:30:18 UTC] USER=www-data EUID=0 PID=1334080 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:18 UTC] USER=www-data EUID=0 PID=1334097 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:18 UTC] USER=www-data EUID=0 PID=1334118 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:18 UTC] USER=www-data EUID=0 PID=1334136 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:30:18 UTC] USER=www-data EUID=0 PID=1334145 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:30:18 UTC] USER=www-data EUID=0 PID=1334154 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334165 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334175 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334184 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334193 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334202 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334211 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334220 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334229 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334238 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334247 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334257 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334266 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334275 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334303 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334312 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334321 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334336 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334346 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334355 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334364 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334373 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334382 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334391 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334400 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334409 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334419 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334429 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334438 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334447 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334456 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:19 UTC] USER=www-data EUID=0 PID=1334474 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334483 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334492 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334501 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334510 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334519 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334528 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334538 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334548 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334559 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334570 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334579 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334588 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334597 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334615 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334624 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334644 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334653 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334663 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334673 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334682 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334709 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334718 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334730 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:30:20 UTC] USER=www-data EUID=0 PID=1334739 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:21 UTC] USER=www-data EUID=0 PID=1334748 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:30:21 UTC] USER=www-data EUID=0 PID=1334759 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:30:21 UTC] USER=www-data EUID=0 PID=1334768 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:30:21 UTC] USER=www-data EUID=0 PID=1334777 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:30:21 UTC] USER=www-data EUID=0 PID=1334798 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:30:21 UTC] USER=www-data EUID=0 PID=1334816 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:30:21 UTC] USER=www-data EUID=0 PID=1334827 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:30:21 UTC] USER=www-data EUID=0 PID=1334836 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:30:21 UTC] USER=www-data EUID=0 PID=1334845 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:30:21 UTC] USER=www-data EUID=0 PID=1334854 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:30:21 UTC] USER=www-data EUID=0 PID=1334863 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:30:21 UTC] USER=www-data EUID=0 PID=1334872 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:30:21 UTC] USER=www-data EUID=0 PID=1334915 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-01-19 13:30:21 UTC] USER=www-data EUID=0 PID=1334924 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-19 13:30:21 UTC] USER=www-data EUID=0 PID=1334934 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
[2026-01-19 13:30:21 UTC] USER=www-data EUID=0 PID=1334949 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-19 13:30:22 UTC] USER=www-data EUID=0 PID=1334962 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:22 UTC] USER=www-data EUID=0 PID=1334978 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:22 UTC] USER=www-data EUID=0 PID=1334988 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:22 UTC] USER=www-data EUID=0 PID=1334999 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:30:22 UTC] USER=www-data EUID=0 PID=1335021 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:30:22 UTC] USER=www-data EUID=0 PID=1335030 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:22 UTC] USER=www-data EUID=0 PID=1335057 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:30:22 UTC] USER=www-data EUID=0 PID=1335066 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:30:22 UTC] USER=www-data EUID=0 PID=1335075 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:30:22 UTC] USER=www-data EUID=0 PID=1335105 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:30:22 UTC] USER=www-data EUID=0 PID=1335133 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:30:22 UTC] USER=www-data EUID=0 PID=1335142 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:30:22 UTC] USER=www-data EUID=0 PID=1335151 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335160 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335170 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335181 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335219 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335228 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335237 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335246 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335283 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335301 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335310 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335320 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335330 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335339 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335348 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335357 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335375 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335384 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335393 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335402 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335411 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335420 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335429 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335439 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:23 UTC] USER=www-data EUID=0 PID=1335449 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335458 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335467 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335476 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335485 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335494 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335503 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335512 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335521 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335530 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335539 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335560 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335570 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335579 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335588 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335597 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335606 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335615 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335624 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335633 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335671 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335681 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335691 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335709 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335718 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335727 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335736 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335754 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:30:24 UTC] USER=www-data EUID=0 PID=1335763 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:30:25 UTC] USER=www-data EUID=0 PID=1335826 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-19 13:30:25 UTC] USER=www-data EUID=0 PID=1335835 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-19 13:30:25 UTC] USER=www-data EUID=0 PID=1335844 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-19 13:30:25 UTC] USER=www-data EUID=0 PID=1335853 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:30:25 UTC] USER=www-data EUID=0 PID=1335868 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:30:25 UTC] USER=www-data EUID=0 PID=1335906 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:30:25 UTC] USER=www-data EUID=0 PID=1335924 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:30:25 UTC] USER=www-data EUID=0 PID=1335933 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:30:25 UTC] USER=www-data EUID=0 PID=1335948 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1335966 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1335975 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1335984 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1335993 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336002 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336012 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336021 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336031 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336052 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336061 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336087 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336096 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336105 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336114 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336132 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336141 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336150 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336159 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336168 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336177 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336186 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336196 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336206 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336215 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336224 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336233 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336251 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:30:26 UTC] USER=www-data EUID=0 PID=1336265 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336278 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336288 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336297 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336306 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336315 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336325 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336335 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336344 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336353 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336362 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336371 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336380 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336389 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336404 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336419 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336428 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336437 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336446 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336459 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336480 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336500 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336513 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336533 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336542 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336552 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336562 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336571 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336581 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336590 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336599 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336608 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336618 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336631 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336640 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336649 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336667 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:30:27 UTC] USER=www-data EUID=0 PID=1336676 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:30:28 UTC] USER=www-data EUID=0 PID=1336685 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:30:28 UTC] USER=www-data EUID=0 PID=1336694 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:30:28 UTC] USER=www-data EUID=0 PID=1336705 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 📦 02 setup pg instance...
[0;34m[DEADLOCK-PREVENTION][0m Deadlock prevention library loaded
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-standby-01-postgresql environment: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com (10.100.1.233)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.233
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Identifier: worker-01-standby-01
[0;34m[INFO][0m Data dir:   /data/postgresql/17/user-sau-main-dev/worker-01-standby-01
[0;34m[INFO][0m Port:       5432
[0;34m[INFO][0m Hostname:   db-user-sau-main-dev-postgresql-worker-01-standby-01
[2026-01-19 13:30:29 UTC] USER=www-data EUID=0 PID=1336845 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:29 UTC] USER=www-data EUID=0 PID=1336866 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:29 UTC] USER=www-data EUID=0 PID=1336887 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:30 UTC] USER=www-data EUID=0 PID=1336913 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[1;33m[WARN][0m Server certificate not found at /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[0;34m[INFO][0m Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  user-sau-main-dev.fastorder.com
  Alt CN:      user-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-19 13:30:30 UTC] USER=www-data EUID=0 PID=1336962 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:30:30 UTC] USER=www-data EUID=0 PID=1336973 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-19 13:30:30 UTC] USER=www-data EUID=0 PID=1336983 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1336921
[2026-01-19 13:30:30 UTC] USER=www-data EUID=0 PID=1336992 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1336921/ra_root.crt
[2026-01-19 13:30:30 UTC] USER=www-data EUID=0 PID=1337001 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1336921/ra_root.key
[2026-01-19 13:30:30 UTC] USER=www-data EUID=0 PID=1337015 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1336921/ra_root.crt
[2026-01-19 13:30:30 UTC] USER=www-data EUID=0 PID=1337027 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1336921/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = user-sau-main-dev.fastorder.com
[2026-01-19 13:30:31 UTC] USER=www-data EUID=0 PID=1337104 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-19 13:30:31 UTC] USER=www-data EUID=0 PID=1337114 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1336921/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:32 UTC] USER=www-data EUID=0 PID=1337123 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:32 UTC] USER=www-data EUID=0 PID=1337132 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-19 13:30:32 UTC] USER=www-data EUID=0 PID=1337141 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-19 13:30:32 UTC] USER=www-data EUID=0 PID=1337152 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
[2026-01-19 13:30:32 UTC] USER=www-data EUID=0 PID=1337161 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
[2026-01-19 13:30:32 UTC] USER=www-data EUID=0 PID=1337170 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-19 13:30:32 UTC] USER=www-data EUID=0 PID=1337179 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-19 13:30:32 UTC] USER=www-data EUID=0 PID=1337188 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = user-sau-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:user-sau-main-dev.fastorder.com, DNS:user-sau-main-dev.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: user-sau-main-dev
Node:        worker-01-standby-01
Primary CN:  user-sau-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=user-sau-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-19 13:30:32 UTC] USER=www-data EUID=0 PID=1337229 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-19 13:30:32 UTC] USER=www-data EUID=0 PID=1337251 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
[2026-01-19 13:30:32 UTC] USER=www-data EUID=0 PID=1337260 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt
[0;32m[OK][0m   mTLS certificates OK (server cert + client certs verified) and keys secured
[0;34m[INFO][0m Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-19 13:30:32 UTC] USER=www-data EUID=0 PID=1337281 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-worker-01-standby-01.service
[2026-01-19 13:30:32 UTC] USER=www-data EUID=0 PID=1337311 ACTION=passthru ARGS=systemctl stop postgresql
[1;33m[WARN][0m Cleaning stale socket directory /var/run/postgresql-user-sau-main-dev-worker-01-standby-01
[2026-01-19 13:30:32 UTC] USER=www-data EUID=0 PID=1337352 ACTION=fsop ARGS=rm -rf /var/run/postgresql-user-sau-main-dev-worker-01-standby-01
[0;32m[OK][0m   No conflicting Postgres left on port 5432
[0;32m[OK][0m   Generated new postgres password for initdb
[2026-01-19 13:30:55 UTC] USER=www-data EUID=0 PID=1337940 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.dCP3wI
[2026-01-19 13:30:55 UTC] USER=www-data EUID=0 PID=1337981 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.dCP3wI
[2026-01-19 13:30:55 UTC] USER=www-data EUID=0 PID=1338018 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/user-sau-main-dev
[2026-01-19 13:30:55 UTC] USER=www-data EUID=0 PID=1338044 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev
[2026-01-19 13:30:55 UTC] USER=www-data EUID=0 PID=1338066 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/user-sau-main-dev
[0;34m[INFO][0m This is a standby. Using pg_basebackup from primary (worker-01)...
[0;34m[INFO][0m Setting up replicator role and slot on primary (worker-01)...
ℹ️  Scanning primary for stuck queries from previous failed attempts...
ℹ️  Scanning for stuck queries (timeout: 30s)...
ℹ️  No stuck queries found
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : user-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SLOT        : worker_01_standby_01
  SSL DIR     : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
  DNS → 10.100.1.232
  CA         : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Role replicator already exists, updating password and ensuring REPLICATION privilege
SET
ALTER ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/user/sau/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🔧 Ensuring replication slot: worker_01_standby_01…
🆕 Creating replication slot worker_01_standby_01
SET
 pg_create_physical_replication_slot 
-------------------------------------
 (worker_01_standby_01,)
(1 row)

✅ Replication slot worker_01_standby_01 created.
🎉 Done.
[0;32m[OK][0m   Replicator role and slot created on primary
[0;34m[INFO][0m Creating replicator client certificates for connecting to primary (worker-01)...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:30:59 UTC] USER=www-data EUID=0 PID=1338358 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-19 13:30:59 UTC] USER=www-data EUID=0 PID=1338376 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-19 13:30:59 UTC] USER=www-data EUID=0 PID=1338388 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-19 13:30:59 UTC] USER=www-data EUID=0 PID=1338398 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-19 13:30:59 UTC] USER=www-data EUID=0 PID=1338407 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:30:59 UTC] USER=www-data EUID=0 PID=1338425 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:30:59 UTC] USER=www-data EUID=0 PID=1338436 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:30:59 UTC] USER=www-data EUID=0 PID=1338446 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338456 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338465 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338476 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338504 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338515 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338525 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338539 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338559 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338568 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338577 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338586 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338595 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338604 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338631 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338640 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338669 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338678 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338687 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338696 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338705 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338723 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338741 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338761 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:31:00 UTC] USER=www-data EUID=0 PID=1338770 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338790 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338800 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338809 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338831 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338851 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338860 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338869 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338878 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338887 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338907 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338917 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338927 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338936 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338954 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338963 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338972 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338981 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338990 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1338999 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1339008 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1339017 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1339026 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:31:01 UTC] USER=www-data EUID=0 PID=1339036 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339056 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339066 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339075 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339085 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339094 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339103 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339112 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339121 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339130 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339139 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339148 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339158 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339169 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339178 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
   ✅ Symlinked ca.pem
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339196 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339214 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339223 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339232 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

[0;32m[OK][0m   Replicator certificate created for worker-01 in /home/postgres/
[0;34m[INFO][0m Using replicator certificates from primary worker-01...
[2026-01-19 13:31:02 UTC] USER=www-data EUID=0 PID=1339314 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt
[0;32m[OK][0m   Replicator certificates verified at /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[0;32m[OK][0m   root.crt verified at /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[0;34m[INFO][0m Updating primary pg_hba.conf to allow replication...
[0;34m[INFO][0m   Standby IP: 10.100.1.233/32 (standby's source IP)
[0;34m[INFO][0m   Primary application IP: 10.100.1.232/32 (for local pg_basebackup)
[0;34m[INFO][0m   Primary DNS IP: 10.100.1.232/32 (DNS resolution of db-user-sau-main-dev-postgresql-worker-01.fastorder.com)
[2026-01-19 13:31:03 UTC] USER=www-data EUID=0 PID=1339423 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.233/32  scram-sha-256 
      $0==begin {inside=1}
      inside && $0==rule {found=1}
      $0==end {inside=0}
      END {exit found?0:1}
     /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[2026-01-19 13:31:03 UTC] USER=www-data EUID=0 PID=1339454 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl  replication  replicator  10.100.1.233/32  scram-sha-256 /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[2026-01-19 13:31:03 UTC] USER=www-data EUID=0 PID=1339484 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.232/32  scram-sha-256 
        $0==begin {inside=1}
        inside && $0==rule {found=1}
        $0==end {inside=0}
        END {exit found?0:1}
       /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[2026-01-19 13:31:03 UTC] USER=www-data EUID=0 PID=1339509 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl  replication  replicator  10.100.1.232/32  scram-sha-256 /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[0;34m[INFO][0m Reloading primary PostgreSQL service...
[2026-01-19 13:31:03 UTC] USER=www-data EUID=0 PID=1339530 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-worker-01.service
[0;32m[OK][0m   Primary pg_hba.conf updated and service reloaded
[1;33m[WARN][0m Removing existing data directory: /data/postgresql/17/user-sau-main-dev/worker-01-standby-01
[2026-01-19 13:31:03 UTC] USER=www-data EUID=0 PID=1339552 ACTION=fsop ARGS=rm -rf /data/postgresql/17/user-sau-main-dev/worker-01-standby-01
[0;34m[INFO][0m Primary host: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[0;34m[INFO][0m Using replicator cert: /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[0;34m[INFO][0m Using replicator key: /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key (PKCS#8 format)
[0;34m[INFO][0m Using CA cert: /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[0;34m[INFO][0m Verifying postgres user can access certificates...
[0;31m[ERR][0m  postgres user CANNOT read /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[0;34m[INFO][0m File permissions:
lrwxrwxrwx 1 postgres ssl-cert 68 Jan 19 13:31 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt -> /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[0;34m[INFO][0m Parent directory permissions:
drwx------ 2 postgres postgres 4096 Jan 19 13:31 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
drwx------ 6 postgres postgres 4096 Jan 18 23:43 /home/postgres/ssl/.postgresql/user-sau-main-dev
[1;33m[WARN][0m Attempting to fix permissions (/usr/local/bin/fastorder-provisioning-wrapper.sh required)...
[0;34m[INFO][0m Fixing /home/postgres/ directory...
[2026-01-19 13:31:04 UTC] USER=www-data EUID=0 PID=1339620 ACTION=fsop ARGS=chmod 755 /home/postgres/
[0;34m[INFO][0m Fixing /home/postgres/ssl/.postgresql/...
[2026-01-19 13:31:04 UTC] USER=www-data EUID=0 PID=1339644 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/
[0;34m[INFO][0m Fixing parent directory: /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:31:04 UTC] USER=www-data EUID=0 PID=1339667 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/user-sau-main-dev
[0;34m[INFO][0m Fixing certificate directory: /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-19 13:31:04 UTC] USER=www-data EUID=0 PID=1339688 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[0;34m[INFO][0m Fixing CA certificate: /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-19 13:31:04 UTC] USER=www-data EUID=0 PID=1339709 ACTION=fsop ARGS=chmod 644 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[0;32m[OK][0m   Permissions fixed
[0;32m[OK][0m   postgres user can now read /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt after permission fix
[2026-01-19 13:31:04 UTC] USER=www-data EUID=0 PID=1339731 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-user-sau-main-dev-worker-01-standby-01
[2026-01-19 13:31:04 UTC] USER=www-data EUID=0 PID=1339752 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-user-sau-main-dev-worker-01-standby-01
[2026-01-19 13:31:04 UTC] USER=www-data EUID=0 PID=1339776 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-user-sau-main-dev-worker-01-standby-01
[0;34m[INFO][0m Checking primary database size before pg_basebackup...
[0;34m[INFO][0m Total primary database size: 29 MB
[0;34m[INFO][0m Estimated transfer time: ~0 minutes (at 10MB/s with compression)
[0;34m[INFO][0m Retrieving replicator password from AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/replicator
[0;32m[OK][0m   Replicator password retrieved successfully
[0;34m[INFO][0m Starting pg_basebackup...
[2026-01-19 13:31:07 UTC] USER=www-data EUID=0 PID=1339876 ACTION=passthru ARGS=sudo -u postgres env PGPASSWORD=4fdUrcEKNirjtl6pfO2YEuBbBDxOb2hE PGSSLMODE=verify-full PGSSLCERT=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt PGSSLKEY=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key PGSSLROOTCERT=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /usr/lib/postgresql/17/bin/pg_basebackup -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -p 5432 -U replicator -D /data/postgresql/17/user-sau-main-dev/worker-01-standby-01 -Fp -Xs -P -R --checkpoint=fast --wal-method=stream --verbose
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/2000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1339887"
30526/30526 kB (100%), 0/1 tablespace (...-01-standby-01/global/pg_control)
30526/30526 kB (100%), 1/1 tablespace                                         
pg_basebackup: write-ahead log end point: 0/2000120
pg_basebackup: waiting for background process to finish streaming ...
pg_basebackup: syncing data to disk ...
pg_basebackup: renaming backup_manifest.tmp to backup_manifest
pg_basebackup: base backup completed
[0;32m[OK][0m   pg_basebackup complete
[0;34m[INFO][0m Fixing postgresql.auto.conf to use IP-based primary_conninfo (matching golden backup)...
[2026-01-19 13:31:07 UTC] USER=www-data EUID=0 PID=1339904 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-19 13:31:07 UTC] USER=www-data EUID=0 PID=1339926 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-19 13:31:07 UTC] USER=www-data EUID=0 PID=1339947 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-19 13:31:07 UTC] USER=www-data EUID=0 PID=1339956 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/standby.signal
[0;32m[OK][0m   standby.signal verified and permissions set
[0;34m[INFO][0m Fixing postgresql.conf with standby-specific settings...
[1;33m[WARN][0m postgresql.conf not found at /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/postgresql.conf
[0;34m[INFO][0m Verifying postgresql.auto.conf...
[1;33m[WARN][0m postgresql.auto.conf not found - pg_basebackup may have failed
[2026-01-19 13:31:07 UTC] USER=www-data EUID=0 PID=1339979 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.dCP3wI
[0;34m[INFO][0m Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[0;32m[OK][0m   postgresql.conf updated successfully
[0;34m[INFO][0m Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-19 13:31:08 UTC] USER=www-data EUID=0 PID=1340030 ACTION=fsop ARGS=cp /tmp/tmp.I9DLtqIgwh /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/pg_hba.conf
[2026-01-19 13:31:08 UTC] USER=www-data EUID=0 PID=1340051 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/pg_hba.conf
[2026-01-19 13:31:08 UTC] USER=www-data EUID=0 PID=1340072 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/pg_hba.conf
[0;32m[OK][0m   pg_hba.conf updated
[0;34m[INFO][0m Creating systemd unit: /etc/systemd/system/postgresql@user-sau-main-dev-worker-01-standby-01.service
[2026-01-19 13:31:08 UTC] USER=www-data EUID=0 PID=1340097 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.BISANv /etc/systemd/system/postgresql@user-sau-main-dev-worker-01-standby-01.service
[2026-01-19 13:31:08 UTC] USER=www-data EUID=0 PID=1340119 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@user-sau-main-dev-worker-01-standby-01.service
[0;32m[OK][0m   systemd unit written
[2026-01-19 13:31:08 UTC] USER=www-data EUID=0 PID=1340140 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-19 13:31:08 UTC] USER=www-data EUID=0 PID=1340161 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-19 13:31:08 UTC] USER=www-data EUID=0 PID=1340194 ACTION=passthru ARGS=systemctl daemon-reload
[0;34m[INFO][0m Starting PostgreSQL instance...
[2026-01-19 13:31:09 UTC] USER=www-data EUID=0 PID=1340342 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-worker-01-standby-01.service
[0;34m[INFO][0m Waiting for ACTIVE (systemd)…
[2026-01-19 13:31:10 UTC] USER=www-data EUID=0 PID=1340386 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-worker-01-standby-01.service
[0;32m[OK][0m   Service ACTIVE
[0;34m[INFO][0m Waiting for port 5432 bind…
[0;32m[OK][0m   Port bound
[0;34m[INFO][0m Waiting pg_isready (socket)…
[0;32m[OK][0m   Readiness via socket OK
[0;34m[INFO][0m Waiting pg_isready (TCP db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com:5432)…
[0;32m[OK][0m   Startup sequence complete
[0;34m[INFO][0m Configuring synchronous replication on primary worker-01...
[0;34m[INFO][0m Current synchronous_standby_names: ''
[0;34m[INFO][0m Initializing synchronous_standby_names with first standby
[0;34m[INFO][0m New synchronous_standby_names: 'ANY 1 (worker_01_standby_01)'
[2026-01-19 13:31:10 UTC] USER=www-data EUID=0 PID=1340474 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_commit = on;
ALTER SYSTEM
[2026-01-19 13:31:10 UTC] USER=www-data EUID=0 PID=1340525 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_standby_names = 'ANY 1 (worker_01_standby_01)';
ALTER SYSTEM
[2026-01-19 13:31:11 UTC] USER=www-data EUID=0 PID=1340549 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[0;32m[OK][0m   ✅ Synchronous replication configured on primary
[0;32m[OK][0m      Setting: ANY 1 (worker_01_standby_01)
[0;34m[INFO][0m Validating core security GUCs (via local socket)…
[0;32m[OK][0m   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[0;34m[INFO][0m Skipping database/role provisioning on standby node (read-only)
[0;34m[INFO][0m   Database/roles will be replicated from primary: worker-01
[0;34m[INFO][0m Applying connection and memory optimizations...
[0;34m[INFO][0m Standby will use primary's max_connections: 100
[0;34m[INFO][0m Current settings: max_connections=100, work_mem=8MB
[0;34m[INFO][0m Target settings (standby): max_connections=100, work_mem=8MB
[0;32m[OK][0m   Connection settings already optimized
[0;34m[INFO][0m Skipping password setting - this is a standby (read-only)
[0;34m[INFO][0m Use primary's postgres password to connect to this standby
[0;34m[INFO][0m Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Identifier: worker-01-standby-01
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.233
[0;34m[INFO][0m Primary hostname: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

[0;34m[INFO][0m Adding /etc/hosts entry for worker-01-standby-01...
[0;34m[INFO][0m   db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.233

[0;34m[INFO][0m   ✅ db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com already exists with correct IP

[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;32m✅[0m   ✅ Network & DNS configuration complete
[0;32m✅[0m   ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
  10.100.1.233    db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com


[0;32m[OK][0m   PostgreSQL 'user-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key \
        host=db-user-sau-main-dev-postgresql-worker-01-standby-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[0;34m[INFO][0m Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        user-sau-main-dev-postgresql-worker-01-standby-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.233
[INFO]   Port:              5432
[INFO]   FQDN:              db-user-sau-main-dev-postgresql-worker-01-standby-01
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 6b53354f-af0c-46ce-9112-1ad9eae0ff4a
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[0;32m[OK][0m   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 📦 03 role...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-19 13:31:16 UTC] USER=www-data EUID=0 PID=1340978 ACTION=fsop ARGS=test -f /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/standby.signal
⚠ This is a PostgreSQL STANDBY (read-only replica)
⚠ Skipping role creation - standby gets roles from primary via replication
⚠ Use the PRIMARY's credentials to connect to this standby


[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (user) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=user
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/user/run.sh
🔍 DEBUG_CHECKPOINT_03: No specific folder for user, using default
[DEBUG] Tracking substep start: steps/01-install/steps/default (RUN_UUID=637d196f-a7be-4345-870b-fb5a079a6ba4)
[INFO] 🔸 Service: user (using default contracts schema)
🔍 DEBUG_CHECKPOINT_04: Executing default: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/run.sh
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting default contracts schema provisioning for SERVICE=user
[INFO] Environment: user-sau-main-dev
[INFO] Schema: user (contracts tables)
[INFO] Identifier: worker-01-standby-01
[INFO] VM IP: 142.93.238.16

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Skipping Schema Setup on worker-01-standby-01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

ℹ️  Schema setup only runs on coordinator
ℹ️  This is a worker-01-standby-01 node - schemas replicate automatically

✅ Nothing to do on this node

[0;32m✓[0m ✅ Standby worker-01-standby-01 setup completed

[0;32m✓[0m ✅ PostgreSQL installation completed
[0;34m[INFO][0m Discovering additional setup steps...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 02-pg-bouncer.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up PgBouncer connection pooling...
[2026-01-19 13:31:21 UTC] USER=www-data EUID=0 PID=1341218 ACTION=fsop ARGS=rm -f /tmp/pgbouncer-ip.service /tmp/pgbouncer.service
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[0;32m✓ [SECRETS][0m Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[0;34m[SECRETS][0m Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[0;34m[SECRETS][0m            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[0;34m[SECRETS][0m            Backups (build_backup_path)
[0;34m[SECRETS][0m Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[0;34m[INFO][0m Checking for existing PgBouncer application environment in topology …
[0;32m[OK][0m   Using existing PgBouncer environment:
[0;34m[INFO][0m   IP:     10.100.1.184
[0;34m[INFO][0m   FQDN:   db-user-sau-main-dev-postgresql-bouncer.fastorder.com
[0;34m[INFO][0m   Domain: db-user-sau-main-dev-postgresql-bouncer.fastorder.com
[0;34m[INFO][0m Ensuring /etc/hosts entry for db-user-sau-main-dev-postgresql-bouncer.fastorder.com …
[0;32m[OK][0m   /etc/hosts already contains entry for db-user-sau-main-dev-postgresql-bouncer.fastorder.com
[1;33m[WARN][0m IP 10.100.1.184 is assigned to multiple interfaces:
    inet 10.100.1.217/32 scope global lo
       valid_lft forever preferred_lft forever
    inet 10.100.1.184/32 scope global lo
--
    inet 10.100.1.219/32 scope global eth0:219
       valid_lft forever preferred_lft forever
    inet 10.100.1.184/32 scope global eth0
[1;33m[WARN][0m This may cause routing issues
[0;34m[INFO][0m Final verification of /etc/hosts entry for db-user-sau-main-dev-postgresql-bouncer.fastorder.com …
[0;32m[OK][0m   /etc/hosts correctly maps db-user-sau-main-dev-postgresql-bouncer.fastorder.com to 10.100.1.184
[1;33m[WARN][0m IP 10.100.1.184 is already bound to other interface(s):
        inet 10.100.1.184/32 scope global lo
        inet 10.100.1.184/32 scope global eth0
[0;34m[INFO][0m Attempting to also bind 10.100.1.184 to lo:pgbouncer ...
[2026-01-19 13:31:22 UTC] USER=www-data EUID=0 PID=1341312 ACTION=passthru ARGS=ip addr add 10.100.1.184/32 dev lo label lo:pgbouncer
RTNETLINK answers: File exists
[0;32m[OK][0m   IP 10.100.1.184 is already bound to lo (may have different label)
[2026-01-19 13:31:22 UTC] USER=www-data EUID=0 PID=1341332 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-19 13:31:23 UTC] USER=www-data EUID=0 PID=1341444 ACTION=passthru ARGS=systemctl restart pgbouncer-ip@user-sau-main-dev.service
[2026-01-19 13:31:23 UTC] USER=www-data EUID=0 PID=1341454 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer-ip@user-sau-main-dev.service
[1;33m[WARN][0m pgbouncer-ip@user-sau-main-dev.service is not active
[1;33m[WARN][0m Check status: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@user-sau-main-dev.service
[2026-01-19 13:31:23 UTC] USER=www-data EUID=0 PID=1341479 ACTION=fsop ARGS=mkdir -p /etc/pgbouncer/user-sau-main-dev
[2026-01-19 13:31:23 UTC] USER=www-data EUID=0 PID=1341488 ACTION=fsop ARGS=mkdir -p /run/pgbouncer/user-sau-main-dev
[2026-01-19 13:31:23 UTC] USER=www-data EUID=0 PID=1341497 ACTION=fsop ARGS=mkdir -p /var/log/pgbouncer/user-sau-main-dev
[2026-01-19 13:31:23 UTC] USER=www-data EUID=0 PID=1341536 ACTION=fsop ARGS=chmod 750 /run/pgbouncer/user-sau-main-dev
[2026-01-19 13:31:23 UTC] USER=www-data EUID=0 PID=1341554 ACTION=fsop ARGS=chmod 750 /var/log/pgbouncer/user-sau-main-dev
[2026-01-19 13:31:23 UTC] USER=www-data EUID=0 PID=1341574 ACTION=fsop ARGS=chown root:postgres /etc/pgbouncer/user-sau-main-dev
[2026-01-19 13:31:23 UTC] USER=www-data EUID=0 PID=1341584 ACTION=fsop ARGS=chown postgres:postgres /run/pgbouncer/user-sau-main-dev
[2026-01-19 13:31:23 UTC] USER=www-data EUID=0 PID=1341607 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbouncer/user-sau-main-dev
[0;34m[INFO][0m Generating pgbouncer_admin client certificates...
[0;34m[INFO][0m ⏳ This may take 30-60 seconds...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username:    pgbouncer_admin
Identifier:  pgbouncer
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Service:     user
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        pgbouncer
  User (CN):   pgbouncer_admin
  Hostname:    db-user-sau-main-dev-postgresql-bouncer.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:31:24 UTC] USER=www-data EUID=0 PID=1341644 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-pgbouncer-pgbouncer_admin
[2026-01-19 13:31:24 UTC] USER=www-data EUID=0 PID=1341653 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-19 13:31:24 UTC] USER=www-data EUID=0 PID=1341662 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
[2026-01-19 13:31:24 UTC] USER=www-data EUID=0 PID=1341672 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-19 13:31:24 UTC] USER=www-data EUID=0 PID=1341681 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = pgbouncer_admin
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:24 UTC] USER=www-data EUID=0 PID=1341700 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:24 UTC] USER=www-data EUID=0 PID=1341709 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:24 UTC] USER=www-data EUID=0 PID=1341718 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-19 13:31:24 UTC] USER=www-data EUID=0 PID=1341727 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-19 13:31:24 UTC] USER=www-data EUID=0 PID=1341738 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341747 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341756 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341766 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341775 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341785 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341794 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341823 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341832 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341841 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341850 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341859 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341868 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341877 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341886 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341912 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341921 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341930 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341948 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341957 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341966 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341975 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341984 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
[2026-01-19 13:31:25 UTC] USER=www-data EUID=0 PID=1341993 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342003 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342032 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342042 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342051 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342076 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342087 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342098 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342107 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342116 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342125 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342134 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342143 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342154 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342164 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342174 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342183 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342192 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342208 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342217 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:26 UTC] USER=www-data EUID=0 PID=1342226 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342244 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342253 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342263 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342274 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342283 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342294 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342317 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342326 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342335 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342344 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342353 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342362 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342371 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342381 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342390 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342400 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342409 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-19 13:31:27 UTC] USER=www-data EUID=0 PID=1342419 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342429 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342442 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342459 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342469 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342478 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342487 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342496 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342505 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342514 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: user-sau-main-dev
User: pgbouncer_admin
Node: pgbouncer
FQDN: db-user-sau-main-dev-postgresql-bouncer.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-user-sau-main-dev-postgresql-bouncer.fastorder.com -U pgbouncer_admin -d postgres

[0;32m[OK][0m   mTLS client certificate present: /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[0;34m[INFO][0m Creating symlinks to canonical certificates in /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend...
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342530 ACTION=fsop ARGS=mkdir -p /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342539 ACTION=fsop ARGS=mkdir -p /etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342548 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342557 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342566 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/root.crt
[0;34m[INFO][0m Creating coordinator CA symlink for PostgreSQL server verification...
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342575 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m Verifying canonical certificate permissions...
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342584 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342611 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[0;32m[OK][0m   Backend certificate symlinks created in /etc/ssl
[0;32m[OK][0m   Coordinator CA symlink created for server verification
[0;32m[OK][0m   Certificates already in canonical location - no symlinks needed
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342633 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/server.key
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342642 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342651 ACTION=fsop ARGS=test -r /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m PgBouncer will use PostgreSQL coordinator CA: /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;32m[OK][0m   PostgreSQL coordinator at db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432 is reachable
[0;34m[INFO][0m Dumping SCRAM secrets from coordinator for PgBouncer auth_file …
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342684 ACTION=fsop ARGS=cp /tmp/tmp.hCH0SDbkQA /etc/pgbouncer/user-sau-main-dev/userlist.txt
[2026-01-19 13:31:28 UTC] USER=www-data EUID=0 PID=1342693 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/user-sau-main-dev/userlist.txt
[2026-01-19 13:31:29 UTC] USER=www-data EUID=0 PID=1342702 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/user-sau-main-dev/userlist.txt
[0;32m[OK][0m   Auth file written: /etc/pgbouncer/user-sau-main-dev/userlist.txt
[0;34m[INFO][0m Retrieved password from vault for pgbouncer_admin
[0;34m[INFO][0m Ensuring PgBouncer admin role 'pgbouncer_admin' exists in Postgres (coordinator) …
[0;32m[OK][0m   Role pgbouncer_admin created/updated successfully
[0;34m[SECRETS][0m Setting credentials in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/pgbouncer_admin
[0;32m✓ [SECRETS][0m Credentials updated in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/pgbouncer_admin
[0;34m[INFO][0m ✅ PgBouncer admin password stored in centralized secrets vault
[0;34m[INFO][0m Re-fetching SCRAM secrets after role creation to ensure pgbouncer_admin is included …
[2026-01-19 13:31:35 UTC] USER=www-data EUID=0 PID=1342850 ACTION=fsop ARGS=cp /tmp/tmp.gcf3Cl8cBB /etc/pgbouncer/user-sau-main-dev/userlist.txt
[2026-01-19 13:31:35 UTC] USER=www-data EUID=0 PID=1342861 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/user-sau-main-dev/userlist.txt
[2026-01-19 13:31:35 UTC] USER=www-data EUID=0 PID=1342870 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/user-sau-main-dev/userlist.txt
[0;32m[OK][0m   Auth file updated with pgbouncer_admin SCRAM hash
[0;34m[INFO][0m Auth file contains [2026-01-19 13:31:35 UTC] USER=www-data EUID=0 PID=1342880 ACTION=passthru ARGS=bash -c wc -l < '/etc/pgbouncer/user-sau-main-dev/userlist.txt'
4 user(s)
[0;32m[OK][0m   Admin 'pgbouncer_admin' password generated and saved
[0;34m[INFO][0m Configuring PostgreSQL to prevent Citus metadata sync hangs...
ALTER ROLE
[0;32m[OK][0m   Disabled Citus metadata sync for pgbouncer_admin
[0;34m[INFO][0m Verifying application database fastorder_user_sau_main_dev_db exists...
[0;32m[OK][0m   ✓ Database fastorder_user_sau_main_dev_db exists
[0;34m[INFO][0m Granting permissions to pgbouncer_admin on fastorder_user_sau_main_dev_db...
GRANT
[0;32m[OK][0m   ✓ Granted CONNECT on fastorder_user_sau_main_dev_db to pgbouncer_admin
GRANT
[0;32m[OK][0m   ✓ Granted USAGE on schema public to pgbouncer_admin
GRANT
[0;32m[OK][0m   ✓ Granted SELECT on all tables to pgbouncer_admin
ALTER DATABASE
[0;32m[OK][0m   Set synchronous_commit=local for fastorder_user_sau_main_dev_db
[0;34m[INFO][0m Ensuring pg_hba.conf entry for pgbouncer_admin …
[0;34m[INFO][0m Adding pg_hba.conf entries for pgbouncer_admin with cert auth …
[0;32m[OK][0m   pg_hba.conf updated and PostgreSQL configuration reloaded
[2026-01-19 13:31:35 UTC] USER=unknown EUID=33 PID=1342925 ACTION=-u ARGS=postgres bash
ERROR: Invalid or unauthorized action: -u
[1;33m[WARN][0m pg_hba.conf entry may not have loaded correctly
[0;34m[INFO][0m Writing /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini …
[2026-01-19 13:31:37 UTC] USER=www-data EUID=0 PID=1342962 ACTION=fsop ARGS=cp /tmp/tmp.b4Dy6Nb1BB /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini
[2026-01-19 13:31:37 UTC] USER=www-data EUID=0 PID=1342971 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini
[2026-01-19 13:31:37 UTC] USER=www-data EUID=0 PID=1342989 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbouncer/user-sau-main-dev /run/pgbouncer/user-sau-main-dev /var/log/pgbouncer/user-sau-main-dev
[2026-01-19 13:31:37 UTC] USER=www-data EUID=0 PID=1342998 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/user-sau-main-dev/userlist.txt
[0;32m[OK][0m   pgbouncer.ini ready
[0;34m[INFO][0m Verifying TLS settings in pgbouncer.ini:
[2026-01-19 13:31:37 UTC] USER=www-data EUID=0 PID=1343008 ACTION=fsop ARGS=grep -E (client_tls_sslmode|server_tls) /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini
client_tls_sslmode = verify-full
server_tls_sslmode = verify-full
server_tls_ca_file = /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
server_tls_cert_file = /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
server_tls_key_file  = /etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;34m[INFO][0m Verifying PgBouncer server certificate files:
[2026-01-19 13:31:37 UTC] USER=www-data EUID=0 PID=1343017 ACTION=fsop ARGS=test -r /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[0;32m[OK][0m   Server cert readable by postgres: /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[0;32m[OK][0m   Server key readable by postgres: /etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;34m[INFO][0m Verifying coordinator CA certificate:
[2026-01-19 13:31:37 UTC] USER=www-data EUID=0 PID=1343035 ACTION=fsop ARGS=test -r /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;32m[OK][0m   Coordinator CA readable by postgres: /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m Preflight: stopping any conflicting PgBouncer on 6432 …
[2026-01-19 13:31:37 UTC] USER=www-data EUID=0 PID=1343044 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer.service
[2026-01-19 13:31:37 UTC] USER=www-data EUID=0 PID=1343053 ACTION=passthru ARGS=systemctl stop pgbouncer@user-sau-main-dev.service
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json?all=1": dial unix /var/run/docker.sock: connect: permission denied
[1;33m[WARN][0m Killing existing pgbouncer processes: 4073760
[2026-01-19 13:33:07 UTC] USER=www-data EUID=0 PID=1345379 ACTION=passthru ARGS=bash -c kill -9 4073760
[2026-01-19 13:33:09 UTC] USER=www-data EUID=0 PID=1345440 ACTION=passthru ARGS=systemctl daemon-reload
[0;32m[OK][0m   systemd unit installed: pgbouncer@user-sau-main-dev.service
[0;34m[INFO][0m Running pre-flight IP conflict check for 10.100.1.184:6432 …
[1;33m[WARN][0m IP conflict checker not found at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/check-ip-conflicts.sh
[1;33m[WARN][0m Skipping pre-flight check - conflicts may occur
[0;34m[INFO][0m Starting PgBouncer (user-sau-main-dev) …
[2026-01-19 13:33:10 UTC] USER=www-data EUID=0 PID=1345538 ACTION=passthru ARGS=systemctl restart pgbouncer@user-sau-main-dev.service
[2026-01-19 13:33:10 UTC] USER=www-data EUID=0 PID=1345549 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer@user-sau-main-dev.service
[0;32m[OK][0m   Service ACTIVE
[0;34m[INFO][0m Verifying auth_file before probing …
[0;34m[INFO][0m Auth file contains 4 user(s)
[1;33m[WARN][0m Auth file does NOT contain pgbouncer_admin entry - authentication will fail
[0;34m[INFO][0m Probing admin console via SSL (psql to database 'pgbouncer') …
[0;34m[INFO][0m Retrieved password from vault for admin console probe
[1;33m[WARN][0m Admin console probe failed (see error below)
psql: error: connection to server at "10.100.1.184", port 6432 failed: server certificate for "db-user-sau-main-dev-postgresql-bouncer.fastorder.com" (and 6 other names) does not match host name "10.100.1.184"
[1;33m[WARN][0m Troubleshooting:
[1;33m[WARN][0m   1. Check auth_file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/user-sau-main-dev/userlist.txt
[1;33m[WARN][0m   2. Test with: PGPASSWORD='yvonAdiGcvLlur+JNgqyr7ru' psql -h 10.100.1.184 -p 6432 -U pgbouncer_admin -d pgbouncer
[1;33m[WARN][0m   3. Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@user-sau-main-dev.service -n 50

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m   Running Comprehensive PgBouncer Verification Tests
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[0;34m[INFO][0m Password extracted: yvonAdiGcv... (using postgres user certificates)

[0;34m[INFO][0m Test 1/7: Admin Console - SHOW POOLS
 database  |   user    | cl_active | cl_waiting | cl_active_cancel_req | cl_waiting_cancel_req | sv_active | sv_active_cancel | sv_being_canceled | sv_idle | sv_used | sv_tested | sv_login | maxwait | maxwait_us | pool_mode | load_balance_hosts 
-----------+-----------+-----------+------------+----------------------+-----------------------+-----------+------------------+-------------------+---------+---------+-----------+----------+---------+------------+-----------+--------------------
 pgbouncer | pgbouncer |         2 |          0 |                    0 |                     0 |         0 |                0 |                 0 |       0 |       0 |         0 |        0 |       0 |          0 | statement | 
(1 row)

[0;32m[OK][0m   ✓ SHOW POOLS: SUCCESS

[0;34m[INFO][0m Test 2/7: Admin Console - SHOW VERSION
[0;32m[OK][0m   ✓ SHOW VERSION: PgBouncer 1.24.1

[0;34m[INFO][0m Test 3/7: Admin Console - SHOW STATS
 database  | total_server_assignment_count | total_xact_count | total_query_count | total_received | total_sent | total_xact_time | total_query_time | total_wait_time | total_client_parse_count | total_server_parse_count | total_bind_count | avg_server_assignment_count | avg_xact_count | avg_query_count | avg_recv | avg_sent | avg_xact_time | avg_query_time | avg_wait_time | avg_client_parse_count | avg_server_parse_count | avg_bind_count 
-----------+-------------------------------+------------------+-------------------+----------------+------------+-----------------+------------------+-----------------+--------------------------+--------------------------+------------------+-----------------------------+----------------+-----------------+----------+----------+---------------+----------------+---------------+------------------------+------------------------+----------------
 pgbouncer |                             0 |                4 |                 4 |              0 |          0 |               0 |                0 |               0 |                        0 |                        0 |                0 |                           0 |              0 |               0 |        0 |        0 |             0 |              0 |             0 |                      0 |                      0 |              0
(1 row)

[0;32m[OK][0m   ✓ SHOW STATS: SUCCESS

[0;34m[INFO][0m Test 4/7: Admin Console - SHOW DATABASES
              name              |                           host                            | port |            database            | force_user | pool_size | min_pool_size | reserve_pool_size | server_lifetime | pool_mode | load_balance_hosts | max_connections | current_connections | max_client_connections | current_client_connections | paused | disabled 
--------------------------------+-----------------------------------------------------------+------+--------------------------------+------------+-----------+---------------+-------------------+-----------------+-----------+--------------------+-----------------+---------------------+------------------------+----------------------------+--------+----------
 fastorder_user_sau_main_dev_db | db-user-sau-main-dev-postgresql-coordinator.fastorder.com | 5432 | fastorder_user_sau_main_dev_db |            |       100 |             0 |                20 |            3600 |           |                    |               0 |                   0 |                      0 |                          0 |      0 |        0
 pgbouncer                      |                                                           | 6432 | pgbouncer                      | pgbouncer  |         2 |             0 |                 0 |            3600 | statement |                    |               0 |                   0 |                      0 |                          2 |      0 |        0
(2 rows)

[0;32m[OK][0m   ✓ SHOW DATABASES: SUCCESS

[0;34m[INFO][0m Test 5/7: Admin Console - SHOW CONFIG
[0;32m[OK][0m   ✓ SHOW CONFIG: SUCCESS
[0;34m[INFO][0m   Key settings:
[0;34m[INFO][0m     client_tls_sslmode = verify-full|disable|yes
[0;34m[INFO][0m     max_client_conn = 2048|100|yes
[0;34m[INFO][0m     pool_mode = transaction|session|yes
[0;34m[INFO][0m     server_tls_sslmode = verify-full|prefer|yes
psql   "host=db-user-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_user_sau_main_dev_db user=pgbouncer_admin password=yvonAdiGcvLlur+JNgqyr7ru    connect_timeout=5 sslmode=verify-full    sslrootcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt    sslcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt    sslkey=/etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key"   --no-psqlrc -Atc 'SELECT version();'

[0;34m[INFO][0m Test 6/7: Application Database - SELECT version()
[1;33m[WARN][0m ✗ Application database query: FAILED (timeout or connection issue)
[1;33m[WARN][0m    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[0;34m[INFO][0m Test 7/8: Application Database - Connection Details
[1;33m[WARN][0m ✗ Connection details: FAILED (timeout or connection issue)
[1;33m[WARN][0m    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[0;34m[INFO][0m Test 8/8: End-to-End Application Routing - Pool Verification
[0;34m[INFO][0m   Running actual queries through PgBouncer to verify routing and pooling...
[1;33m[WARN][0m ✗ End-to-end routing verification: FAILED - All 3 queries failed
[1;33m[WARN][0m    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[1;33m[WARN][0m    Otherwise check if database fastorder_user_sau_main_dev_db exists and user pgbouncer_admin has permissions

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m   Verification Complete - Tests 1-5 PASSED (Admin console verified)
[1;33m[WARN][0m   Tests 6-8 FAILED - Application database not accessible
[1;33m[WARN][0m   This is expected if Citus is not set up yet
[1;33m[WARN][0m   Run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[0;32m[OK][0m   PgBouncer is up for user-sau-main-dev

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Connection Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Password stored in: AWS Secrets Manager (fastorder/db/web/ksa/main/dev/postgresqluser/sau/main/dev/coordinator-pgbouncer_admin)
Current password: yvonAdiGcvLlur+JNgqyr7ru

1. Admin Console (using IP address to avoid DNS/SSL issues):
   psql "host=10.100.1.184 port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=yvonAdiGcvLlur+JNgqyr7ru sslmode=verify-full sslrootcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

2. Admin Console (using hostname):
   psql "host=db-user-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=yvonAdiGcvLlur+JNgqyr7ru sslmode=verify-full sslrootcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

3. Application Database:
   psql "host=db-user-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_user_sau_main_dev_db sslkey=/etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=yvonAdiGcvLlur+JNgqyr7ru sslmode=verify-full sslrootcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

4. Using .pgpass file:
   echo "db-user-sau-main-dev-postgresql-bouncer.fastorder.com:6432:*:pgbouncer_admin:yvonAdiGcvLlur+JNgqyr7ru" >> ~/.pgpass
   chmod 600 ~/.pgpass
   psql -h db-user-sau-main-dev-postgresql-bouncer.fastorder.com -p 6432 -U pgbouncer_admin -d fastorder_user_sau_main_dev_db

5. Retrieve password from vault:
   source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
   PGPASSWORD="$(get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password')" \
     psql -h 10.100.1.184 -p 6432 -U pgbouncer_admin -d pgbouncer -c "SHOW POOLS;"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Architecture
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  • Default db 'fastorder_user_sau_main_dev_db' → Citus coordinator (db-user-sau-main-dev-postgresql-coordinator.fastorder.com)
  • Worker access: 'fastorder_user_sau_main_dev_db_worker_1', 'fastorder_user_sau_main_dev_db_worker_2', … (if exist)
  • Client TLS: require (password auth) / verify-full (mTLS with certs)
  • Server TLS: verify-full (PgBouncer validates PostgreSQL certs)
  • Auth: SCRAM-SHA-256 via /etc/pgbouncer/user-sau-main-dev/userlist.txt
  • Pool mode: transaction (stateless connections)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Management
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Service Status:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer@user-sau-main-dev.service
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@user-sau-main-dev.service

Logs:
  command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@user-sau-main-dev.service -f
  /usr/local/bin/fastorder-provisioning-wrapper.sh tail -f /var/log/pgbouncer/user-sau-main-dev/pgbouncer.log

Reload Config:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@user-sau-main-dev.service

Restart:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@user-sau-main-dev.service

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Files
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Config:        /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini
Auth file:     /etc/pgbouncer/user-sau-main-dev/userlist.txt
Server cert:   /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/server.crt
Server key:    /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/server.key
CA cert:       /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt
PG CA:         /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
Logs:          /var/log/pgbouncer/user-sau-main-dev/pgbouncer.log

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Troubleshooting
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


If "SASL authentication failed":
  1. Check auth file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/user-sau-main-dev/userlist.txt
  2. Verify pgbouncer_admin is present with SCRAM hash
  3. Get password from vault:
     source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
     get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password'
  4. Reload PgBouncer: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@user-sau-main-dev.service

If "no pg_hba.conf entry":
  1. Check pg_hba.conf on coordinator
  2. Add rule: hostssl all pgbouncer_admin 10.100.1.184/32 cert clientcert=verify-full
  3. Reload PostgreSQL

To add users to PgBouncer:
  1. Create user in PostgreSQL with password
  2. Re-run SCRAM dump:
     psql "host=db-user-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 dbname=postgres user=postgres \
       sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt \
       sslcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt sslkey=/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key" \
       -Atc "SELECT '\"' || rolname || '\" \"' || rolpassword || '\"' \
             FROM pg_authid WHERE rolpassword LIKE 'SCRAM-SHA-256%' \
             AND rolcanlogin ORDER BY rolname;" | command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop tee /etc/pgbouncer/user-sau-main-dev/userlist.txt
  3. Reload: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@user-sau-main-dev.service

[0;34m[INFO][0m Registering PgBouncer node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        user-sau-main-dev-pgbouncer
[INFO]   Identifier Parent: postgresql
[INFO]   IP:                10.100.1.184
[INFO]   Port:              6432
[INFO]   FQDN:              db-user-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: c866fe26-0c2d-4619-a98d-8cd82c922b78
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[0;32m[OK][0m   PgBouncer node registered to observability API
[0;32m✓[0m ✅ PgBouncer setup completed

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 03-citus-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m CITUS DISTRIBUTED CLUSTER SETUP
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Phase 1: Installing Citus extension on workers...
[0;34m[INFO][0m Phase 2: Setting up coordinator and registering workers...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[0;34m[INFO][0m 📦 PHASE 1: Installing Citus extension on 1 worker(s)...

[0;34m[INFO][0m → Worker 1/1: Installing Citus on worker-01...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS CLUSTER SETUP
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════

[0;34m[INFO][0m 🔧 Setting up Citus Worker...
[0;34m[INFO][0m Temporarily disabling synchronous replication for extension installation...
t
[0;34m[INFO][0m Installing Citus extension on worker...
[0;32m[OK][0m   Citus extension installed on worker
[0;34m[INFO][0m Restoring synchronous replication settings...
t
[0;34m[INFO][0m Worker Citus extension installed - registration will happen when coordinator setup runs

[0;32m[OK][0m   Citus setup complete for worker-01
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m✓[0m   ✅ Citus extension installed on worker-01

[0;32m✓[0m ✅ Phase 1 Complete: All 1 workers have Citus extension installed

[0;34m[INFO][0m 🔧 PHASE 2: Setting up Citus coordinator and registering workers...

[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS CLUSTER SETUP
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════

[0;34m[INFO][0m 🔧 Setting up Citus Coordinator...

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m DIAGNOSTIC: Configuration Variables
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m PG_WORKERS_NUM: 1
[0;34m[INFO][0m ENV_ID: user-sau-main-dev
[0;34m[INFO][0m DOMAIN: fastorder.com
[0;34m[INFO][0m PORT: 5432
[0;34m[INFO][0m SOCKET_DIR: /var/run/postgresql-user-sau-main-dev-coordinator
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[0;34m[INFO][0m Ensuring postgres client certificates exist for coordinator...
[0;32m[OK][0m   Postgres client certificates already exist for coordinator
[0;34m[INFO][0m Adding citus_cert_map to coordinator pg_ident.conf...
[0;32m[OK][0m   pg_ident.conf updated for coordinator
[0;34m[INFO][0m Installing Citus extension on coordinator...
[0;32m[OK][0m   Citus extension installed on coordinator (postgres database)
[0;34m[INFO][0m Installing Citus extension on application database: fastorder_user_sau_main_dev_db...
[0;32m[OK][0m   Citus extension installed on application database: fastorder_user_sau_main_dev_db
[0;34m[INFO][0m Configuring Citus SSL connection parameters...
[2026-01-19 13:33:35 UTC] USER=www-data EUID=0 PID=1346439 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-coordinator.service
[0;32m[OK][0m   ✅ Citus SSL connection parameters configured: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[1;33m[WARN][0m Node not identified as coordinator, initializing...
[0;34m[INFO][0m Checking coordinator configuration...
[0;34m[INFO][0m Persisting citus.local_hostname to postgresql.conf...
[2026-01-19 13:33:37 UTC] USER=www-data EUID=0 PID=1346489 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/user-sau-main-dev/coordinator/postgresql.conf
[2026-01-19 13:33:37 UTC] USER=www-data EUID=0 PID=1346512 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-coordinator.service
[0;32m[OK][0m   ✅ citus.local_hostname persisted to config and reloaded
[0;34m[INFO][0m Configuring coordinator hostname in postgres database: db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432

[0;32m[OK][0m   ✅ Coordinator hostname set to db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432 in postgres database
[0;34m[INFO][0m Checking coordinator configuration in application database: fastorder_user_sau_main_dev_db...
[1;33m[WARN][0m ⚠️  Coordinator registered as 'localhost' in application database, fixing...
[0;34m[INFO][0m Configuring coordinator hostname in application database: db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432
[0;32m[OK][0m   ✅ Coordinator hostname set to db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432 in application database
[0;34m[INFO][0m Validating coordinator configuration before worker registration...
[0;32m[OK][0m   ✅ Coordinator hostname validated: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[0;32m[OK][0m   ✅ citus_tables view is accessible
[0;34m[INFO][0m Checking coordinator self-registration...
[0;32m[OK][0m   ✅ Coordinator is already self-registered
[0;34m[INFO][0m Configuring coordinator shard placement policy...
[0;32m[OK][0m   ✅ Coordinator already configured in postgres database (shouldhaveshards = false)
[1;33m[WARN][0m ⚠️  Coordinator has 66 shards in fastorder_user_sau_main_dev_db - cannot set shouldhaveshards=false
[1;33m[WARN][0m    You must rebalance shards to workers first, then run this setup again
[1;33m[WARN][0m    Skipping shouldhaveshards configuration for application database
[0;34m[INFO][0m Registering 1 worker(s) to Citus cluster...

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m PRE-FLIGHT: Checking worker availability...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Checking worker worker-01...
[0;34m[INFO][0m   FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[0;32m[OK][0m   ✅ Worker worker-01 is reachable via SSL
[0;32m[OK][0m   All workers are reachable - proceeding with registration

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Adding Citus worker: db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Adding citus_cert_map to worker-01 pg_ident.conf...
[0;32m[OK][0m   pg_ident.conf updated for worker-01
[0;34m[INFO][0m Configuring worker worker-01 HBA for coordinator (10.100.1.231) access...
[0;32m[OK][0m   Worker worker-01 HBA configured for coordinator (10.100.1.231)
[0;34m[INFO][0m Adding replication rules for 3 standby(s)...
[0;32m[OK][0m   Replication rules already exist for worker-01
[0;34m[INFO][0m Reloading worker worker-01 to apply HBA changes...
[2026-01-19 13:33:40 UTC] USER=www-data EUID=0 PID=1346681 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-worker-01.service
[0;34m[INFO][0m Configuring coordinator HBA for worker worker-01 (10.100.1.232) access...
[0;32m[OK][0m   Coordinator HBA configured for worker worker-01 (10.100.1.232)
[0;34m[INFO][0m Reloading coordinator to apply HBA changes...
[2026-01-19 13:33:41 UTC] USER=www-data EUID=0 PID=1346713 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-coordinator.service
[0;34m[INFO][0m Ensuring postgres client certificates exist for worker-01...
[0;32m[OK][0m   Postgres client certificates already exist for worker-01
[0;34m[INFO][0m Configuring citus.node_conninfo on worker-01...
[2026-01-19 13:33:41 UTC] USER=www-data EUID=0 PID=1346730 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-worker-01.service
[0;32m[OK][0m   citus.node_conninfo configured on worker-01
[0;34m[INFO][0m Temporarily relaxing sync-rep on worker worker-01...
t
[0;32m[OK][0m   Worker worker-01 sync-rep relaxed (was: sync_commit=on)
[0;34m[INFO][0m Ensuring Citus extension on worker databases...
CREATE EXTENSION
CREATE EXTENSION
[0;34m[INFO][0m Running citus_add_node with 180s timeout...
NOTICE:  shards are still on the coordinator after adding the new node
HINT:  Use SELECT rebalance_table_shards(); to balance shards data between workers and coordinator or SELECT citus_drain_node('db-user-sau-main-dev-postgresql-coordinator.fastorder.com',5432); to permanently move shards away from the coordinator.
2
[0;34m[INFO][0m Restoring worker worker-01 sync-rep settings...
t
[0;32m[OK][0m   Worker worker-01 sync-rep restored
[0;32m[OK][0m   ✅ Worker db-user-sau-main-dev-postgresql-worker-01.fastorder.com successfully added to Citus cluster
[0;34m[INFO][0m    Node ID: 2
[0;34m[INFO][0m    Registered in: postgres, fastorder_user_sau_main_dev_db
[0;32m[OK][0m   Worker worker-01 registration successful
[0;34m[INFO][0m Configuring worker worker-01 shard placement policy...
[0;32m[OK][0m   ✅ Worker worker-01 configured to hold shards in all databases


[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m POST-REGISTRATION: Verifying cluster state...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Expected workers: 1
[0;34m[INFO][0m Registered workers: 1
[0;32m[OK][0m   ✅ All 1 workers successfully registered!

[0;34m[INFO][0m Citus cluster configuration:
db-user-sau-main-dev-postgresql-coordinator.fastorder.com  5432  0  t  primary  f
db-user-sau-main-dev-postgresql-worker-01.fastorder.com    5432  1  t  primary  t

[0;34m[INFO][0m Note: groupid=0 is the coordinator, groupid>0 are workers
[0;34m[INFO][0m       shouldhaveshards: false=query router only, true=holds data shards

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m FINAL VALIDATION: Verifying configuration persistence...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:33:44 UTC] USER=www-data EUID=0 PID=1346982 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/user-sau-main-dev/coordinator/postgresql.conf
[0;32m[OK][0m   ✅ citus.local_hostname persisted in postgresql.conf
[0;32m[OK][0m   ✅ All 1 worker(s) successfully registered and verified

[0;32m[OK][0m   ✅ All validation checks passed
[0;32m[OK][0m   Citus coordinator setup complete

[0;32m[OK][0m   Citus setup complete for coordinator
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════

[0;32m✓[0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✓[0m ✅ CITUS CLUSTER SETUP COMPLETED SUCCESSFULLY
[0;32m✓[0m    Coordinator: Ready and accepting connections
[0;32m✓[0m    Workers registered: 1
[0;32m✓[0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 05-backup-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up coordinator backup...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for user-sau-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-01-19 13:33:46 UTC] USER=www-data EUID=0 PID=1347072 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/user-sau-main-dev
[2026-01-19 13:33:46 UTC] USER=www-data EUID=0 PID=1347087 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/user-sau-main-dev
[2026-01-19 13:33:46 UTC] USER=www-data EUID=0 PID=1347096 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-01-19 13:33:46 UTC] USER=www-data EUID=0 PID=1347105 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-19 13:33:46 UTC] USER=www-data EUID=0 PID=1347116 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-19 13:33:46 UTC] USER=www-data EUID=0 PID=1347125 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-19 13:33:50 UTC] USER=www-data EUID=0 PID=1347245 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-19 13:33:50 UTC] USER=www-data EUID=0 PID=1347254 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-19 13:33:51 UTC] USER=www-data EUID=0 PID=1347273 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/user-sau-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-user-sau-main-dev
[2026-01-19 13:33:51 UTC] USER=www-data EUID=0 PID=1347322 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-19 13:33:51 UTC] USER=www-data EUID=0 PID=1347338 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[INFO] Ensuring correct ownership...
[2026-01-19 13:33:51 UTC] USER=www-data EUID=0 PID=1347368 ACTION=fsop ARGS=chown -R postgres:postgres /data/postgresql/17/user-sau-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-19 13:33:51 UTC] USER=www-data EUID=0 PID=1347378 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-01-19 13:33:51 UTC] USER=www-data EUID=0 PID=1347387 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-01-19 13:33:51 UTC] USER=www-data EUID=0 PID=1347402 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-19 13:33:51 UTC] USER=www-data EUID=0 PID=1347416 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/user-sau-main-dev/coordinator/PG_VERSION
[2026-01-19 13:33:51 UTC] USER=www-data EUID=0 PID=1347428 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza user-sau-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-19 13:33:51 UTC] USER=www-data EUID=0 PID=1347503 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-19 13:33:53 UTC] USER=www-data EUID=0 PID=1347549 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-coordinator.service
[2026-01-19 13:33:57 UTC] USER=www-data EUID=0 PID=1347673 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-19 13:33:57 UTC] USER=www-data EUID=0 PID=1347697 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=user-sau-main-dev-coordinator --log-level-console=info check
2026-01-19 13:33:57.933 P00   INFO: check command begin 2.56.0: --exec-id=1347705-09d0a927 --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/user-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-user-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/user-sau-main-dev --stanza=user-sau-main-dev-coordinator
2026-01-19 13:33:57.947 P00   INFO: check repo1 configuration (primary)
2026-01-19 13:33:57.960 P00  ERROR: [028]: backup and archive info files exist but do not match the database
                                    HINT: is this the correct stanza?
                                    HINT: did an error occur during stanza-upgrade?
2026-01-19 13:33:57.960 P00   INFO: check command end: aborted with exception [028]
[WARN] ⚠️  Stanza verification failed - this may be normal if WAL archiving hasn't started yet
[WARN]    The backup system is configured and will work once WAL segments are generated

[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-19 13:33:58 UTC] USER=www-data EUID=0 PID=1347718 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|user-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-user-sau-main-dev.sh
[2026-01-19 13:33:58 UTC] USER=www-data EUID=0 PID=1347727 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-user-sau-main-dev.sh
[2026-01-19 13:33:58 UTC] USER=www-data EUID=0 PID=1347754 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-user-sau-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-19 13:33:58 UTC] USER=www-data EUID=0 PID=1347772 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-user-sau-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-01-19 13:33:58 UTC] USER=www-data EUID=0 PID=1347790 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|user-sau-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-19 13:33:58 UTC] USER=www-data EUID=0 PID=1347799 ACTION=fsop ARGS=sed -i s|__ENV_ID__|user-sau-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-19 13:33:58 UTC] USER=www-data EUID=0 PID=1347817 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-19 13:33:58 UTC] USER=www-data EUID=0 PID=1347826 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-19 13:33:58.701 P00   INFO: start command begin 2.56.0: --exec-id=1347904-08e56c2d --log-level-console=info --log-level-file=debug --stanza=user-sau-main-dev-coordinator
2026-01-19 13:33:58.710 P00   WARN: stop file does not exist for stanza user-sau-main-dev-coordinator
2026-01-19 13:33:58.710 P00   INFO: start command end: completed successfully (20ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-19 13:33:58.793 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=1347964-65d1339b --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/user-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-user-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/user-sau-main-dev --stanza=user-sau-main-dev-coordinator
2026-01-19 13:33:58.796 P00   INFO: stanza-upgrade for stanza 'user-sau-main-dev-coordinator' on repo1
2026-01-19 13:33:58.825 P00   INFO: stanza-upgrade command end: completed successfully (36ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-19 13:33:58 UTC] USER=www-data EUID=0 PID=1347971 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260119-133358.log
[2026-01-19 13:33:58 UTC] USER=www-data EUID=0 PID=1347981 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260119-133358.log
[INFO] Running backup (timeout: 10 minutes)...
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260119-133358.log
   2026-01-19 13:34:07.839 P00   INFO: repo1: remove expired backup 20260118-220155F
   2026-01-19 13:34:07.898 P00   INFO: repo1: 17-22 remove archive, start = 000000010000000000000003, stop = 000000010000000000000005
   2026-01-19 13:34:07.900 P00   INFO: repo1: 17-23 no archive to remove
   2026-01-19 13:34:07.900 P00   INFO: repo1: 17-24 remove archive, start = 000000010000000000000002, stop = 000000010000000000000002
   2026-01-19 13:34:07.900 P00   INFO: expire command end: completed successfully (73ms)

[INFO] Current backups:
stanza: user-sau-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000006/0000000100000000000000AE

        full backup: 20260118-220214F
            timestamp start/stop: 2026-01-18 22:02:14+00 / 2026-01-18 22:02:16+00
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.6MB, backup size: 5.6MB

    db (prior)
        wal archive min/max (17): 000000010000000000000003/00000001000000040000006E

        full backup: 20260118-234609F
            timestamp start/stop: 2026-01-18 23:46:09+00 / 2026-01-18 23:46:17+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.6MB, backup size: 5.6MB

        full backup: 20260118-234628F
            timestamp start/stop: 2026-01-18 23:46:28+00 / 2026-01-18 23:46:34+00
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.6MB, backup size: 5.6MB

        diff backup: 20260118-234628F_20260119-020006D
            timestamp start/stop: 2026-01-19 02:00:06+00 / 2026-01-19 02:00:15+00
            wal start/stop: 0000000100000000000000BF / 0000000100000000000000C9
            database size: 37.7MB, database backup size: 9.2MB
            repo1: backup set size: 5.7MB, backup size: 1.8MB
            backup reference total: 1 full

    db (current)
        wal archive min/max (17): none present

        full backup: 20260119-133359F
            timestamp start/stop: 2026-01-19 13:33:59+00 / 2026-01-19 13:34:07+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.6MB, backup size: 5.6MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         user-sau-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/user-sau-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/user-sau-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=user-sau-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up worker backups for 1 worker(s)...
[0;34m[INFO][0m Setting up backup for: worker-01
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for user-sau-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-01-19 13:34:08 UTC] USER=www-data EUID=0 PID=1348301 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/user-sau-main-dev
[2026-01-19 13:34:08 UTC] USER=www-data EUID=0 PID=1348313 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/user-sau-main-dev
[2026-01-19 13:34:08 UTC] USER=www-data EUID=0 PID=1348332 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-19 13:34:08 UTC] USER=www-data EUID=0 PID=1348343 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-19 13:34:08 UTC] USER=www-data EUID=0 PID=1348352 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-19 13:34:10 UTC] USER=www-data EUID=0 PID=1348390 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-19 13:34:10 UTC] USER=www-data EUID=0 PID=1348399 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-19 13:34:10 UTC] USER=www-data EUID=0 PID=1348408 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-01-19 13:34:10 UTC] USER=www-data EUID=0 PID=1348417 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/user-sau-main-dev
[2026-01-19 13:34:10 UTC] USER=www-data EUID=0 PID=1348426 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/user-sau-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-user-sau-main-dev
[2026-01-19 13:34:10 UTC] USER=www-data EUID=0 PID=1348450 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-19 13:34:10 UTC] USER=www-data EUID=0 PID=1348459 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-01-19 13:34:10 UTC] USER=www-data EUID=0 PID=1348470 ACTION=fsop ARGS=find /data/postgresql/17/user-sau-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-01-19 13:34:10 UTC] USER=www-data EUID=0 PID=1348479 ACTION=fsop ARGS=chown -R postgres:postgres /data/postgresql/17/user-sau-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-19 13:34:10 UTC] USER=www-data EUID=0 PID=1348488 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-01-19 13:34:10 UTC] USER=www-data EUID=0 PID=1348497 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-01-19 13:34:10 UTC] USER=www-data EUID=0 PID=1348506 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-19 13:34:10 UTC] USER=www-data EUID=0 PID=1348515 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/user-sau-main-dev/coordinator/PG_VERSION
[2026-01-19 13:34:10 UTC] USER=www-data EUID=0 PID=1348525 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza user-sau-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-19 13:34:11 UTC] USER=www-data EUID=0 PID=1348612 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-19 13:34:13 UTC] USER=www-data EUID=0 PID=1348697 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-coordinator.service
[2026-01-19 13:34:17 UTC] USER=www-data EUID=0 PID=1348820 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-19 13:34:17 UTC] USER=www-data EUID=0 PID=1348849 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=user-sau-main-dev-coordinator --log-level-console=info check
2026-01-19 13:34:17.591 P00   INFO: check command begin 2.56.0: --exec-id=1348857-2276c2c5 --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/user-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-user-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/user-sau-main-dev --stanza=user-sau-main-dev-coordinator
2026-01-19 13:34:17.611 P00   INFO: check repo1 configuration (primary)
2026-01-19 13:34:17.660 P00   INFO: check repo1 archive for WAL (primary)
2026-01-19 13:34:17.961 P00   INFO: WAL segment 000000010000000000000005 successfully archived to '/var/lib/pgbackrest/backup/user-sau-main-dev/archive/user-sau-main-dev-coordinator/17-24/0000000100000000/000000010000000000000005-f59dd864bb7213d0436c4c164353a93f34620925.lz4' on repo1
2026-01-19 13:34:17.961 P00   INFO: check command end: completed successfully (373ms)
[INFO] ✅ Stanza verification passed

[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-19 13:34:18 UTC] USER=www-data EUID=0 PID=1348879 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|user-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-user-sau-main-dev.sh
[2026-01-19 13:34:18 UTC] USER=www-data EUID=0 PID=1348888 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-user-sau-main-dev.sh
[2026-01-19 13:34:18 UTC] USER=www-data EUID=0 PID=1348907 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|user-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-user-sau-main-dev.sh
[2026-01-19 13:34:18 UTC] USER=www-data EUID=0 PID=1348916 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-user-sau-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-19 13:34:18 UTC] USER=www-data EUID=0 PID=1348934 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-user-sau-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-01-19 13:34:18 UTC] USER=www-data EUID=0 PID=1348952 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|user-sau-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-19 13:34:18 UTC] USER=www-data EUID=0 PID=1348961 ACTION=fsop ARGS=sed -i s|__ENV_ID__|user-sau-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-19 13:34:18 UTC] USER=www-data EUID=0 PID=1348970 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/data/postgresql/17/user-sau-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-19 13:34:18 UTC] USER=www-data EUID=0 PID=1348979 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-19 13:34:18 UTC] USER=www-data EUID=0 PID=1348988 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-19 13:34:18.316 P00   INFO: start command begin 2.56.0: --exec-id=1349010-7cc634d7 --log-level-console=info --log-level-file=debug --stanza=user-sau-main-dev-coordinator
2026-01-19 13:34:18.316 P00   WARN: stop file does not exist for stanza user-sau-main-dev-coordinator
2026-01-19 13:34:18.316 P00   INFO: start command end: completed successfully (3ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-19 13:34:18.355 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=1349021-c3247f47 --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/user-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-user-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/user-sau-main-dev --stanza=user-sau-main-dev-coordinator
2026-01-19 13:34:18.355 P00   INFO: stanza-upgrade for stanza 'user-sau-main-dev-coordinator' on repo1
2026-01-19 13:34:18.356 P00   INFO: stanza 'user-sau-main-dev-coordinator' on repo1 is already up to date
2026-01-19 13:34:18.356 P00   INFO: stanza-upgrade command end: completed successfully (5ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-19 13:34:18 UTC] USER=www-data EUID=0 PID=1349025 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260119-133418.log
[2026-01-19 13:34:18 UTC] USER=www-data EUID=0 PID=1349034 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260119-133418.log
[2026-01-19 13:34:18 UTC] USER=www-data EUID=0 PID=1349043 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260119-133418.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-01-19 13:34:20 UTC] USER=www-data EUID=0 PID=1349124 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-1348275.log /var/log/pgbackrest/initial-backup-20260119-133418.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260119-133418.log
   2026-01-19 13:34:20.779 P00   INFO: repo1: remove expired backup 20260118-220214F
   2026-01-19 13:34:20.816 P00   INFO: repo1: remove archive path /var/lib/pgbackrest/backup/user-sau-main-dev/archive/user-sau-main-dev-coordinator/17-22
   2026-01-19 13:34:20.824 P00   INFO: repo1: 17-23 no archive to remove
   2026-01-19 13:34:20.825 P00   INFO: repo1: 17-24 no archive to remove
   2026-01-19 13:34:20.825 P00   INFO: expire command end: completed successfully (54ms)

[INFO] Current backups:
stanza: user-sau-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000003/00000001000000040000006E

        full backup: 20260118-234609F
            timestamp start/stop: 2026-01-18 23:46:09+00 / 2026-01-18 23:46:17+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.6MB, backup size: 5.6MB

        full backup: 20260118-234628F
            timestamp start/stop: 2026-01-18 23:46:28+00 / 2026-01-18 23:46:34+00
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.6MB, backup size: 5.6MB

        diff backup: 20260118-234628F_20260119-020006D
            timestamp start/stop: 2026-01-19 02:00:06+00 / 2026-01-19 02:00:15+00
            wal start/stop: 0000000100000000000000BF / 0000000100000000000000C9
            database size: 37.7MB, database backup size: 9.2MB
            repo1: backup set size: 5.7MB, backup size: 1.8MB
            backup reference total: 1 full

    db (current)
        wal archive min/max (17): 000000010000000000000003/000000010000000000000006

        full backup: 20260119-133359F
            timestamp start/stop: 2026-01-19 13:33:59+00 / 2026-01-19 13:34:07+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.6MB, backup size: 5.6MB

        full backup: 20260119-133418F
            timestamp start/stop: 2026-01-19 13:34:18+00 / 2026-01-19 13:34:20+00
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.6MB, backup size: 5.6MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         user-sau-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/user-sau-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/user-sau-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=user-sau-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✓[0m ✅ Backup setup completed for coordinator and all workers

[0;34m[INFO][0m Skipping 06-distribute-tables-canary.sh (test script - set RUN_TESTS=true to enable)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 07-distribute-tables.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:34:22 UTC] USER=unknown EUID=33 PID=1349179 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-19 13:34:22 UTC] USER=unknown EUID=33 PID=1349186 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-19 13:34:22 UTC] USER=unknown EUID=33 PID=1349193 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-19 13:34:22 UTC] USER=unknown EUID=33 PID=1349200 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS TABLE DISTRIBUTION
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════

[0;34m[INFO][0m 🔐 Secure connection established
[0;34m[INFO][0m    Host: db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432
[0;34m[INFO][0m    Database: fastorder_user_sau_main_dev_db
[0;34m[INFO][0m    SSL: verify-full (TLS 1.2+)
[0;34m[INFO][0m    Timeouts: statement=120s, idle_tx=300s

[0;34m[INFO][0m 🔍 Running preflight checks...
[0;34m[INFO][0m Testing database connectivity...
[0;32m[OK][0m   ✅ Database connection successful
[0;32m[OK][0m   ✅ Connected to correct database: fastorder_user_sau_main_dev_db
[0;34m[INFO][0m Checking Citus extension in database fastorder_user_sau_main_dev_db...
[0;32m[OK][0m   Citus version: 13.2-1
[0;34m[INFO][0m Checking worker registration...
[0;32m[OK][0m   Registered workers: 1
[0;34m[INFO][0m Worker nodes:
[0;34m[INFO][0m                           nodename                         | nodeport | isactive | noderole 
[0;34m[INFO][0m   ---------------------------------------------------------+----------+----------+----------
[0;34m[INFO][0m    db-user-sau-main-dev-postgresql-worker-01.fastorder.com |     5432 | t        | primary
[0;34m[INFO][0m   (1 row)
[0;34m[INFO][0m   

[0;34m[INFO][0m 📊 Starting table distribution...

[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Distributing: auth.login_account
[0;34m[INFO][0m Description: User authentication table - distributed by region for tenant isolation
[0;34m[INFO][0m Shard key: region_hint
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ⏭️  Table does not exist, skipping

[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m[OK][0m   ✅ All tables distributed successfully!
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════

[0;34m[INFO][0m 📊 Citus Cluster Summary:

[0;34m[INFO][0m Distributed tables:
[0;34m[INFO][0m              table           |    type     | shard_key | shards |  size   
[0;34m[INFO][0m   ---------------------------+-------------+-----------+--------+---------
[0;34m[INFO][0m    "user".contract_key       | reference   | <none>    |      1 | 16 kB
[0;34m[INFO][0m    "user".contract_type      | reference   | <none>    |      1 | 16 kB
[0;34m[INFO][0m    "user".contract_term_json | distributed | id        |     32 | 512 kB
[0;34m[INFO][0m    "user".contract_term_vars | distributed | id        |     32 | 1792 kB
[0;34m[INFO][0m   (4 rows)
[0;34m[INFO][0m   

[0;34m[INFO][0m Worker capacity:
[0;34m[INFO][0m    worker | total_shards | total_size 
[0;34m[INFO][0m   --------+--------------+------------
[0;34m[INFO][0m   (0 rows)
[0;34m[INFO][0m   

[0;32m[OK][0m   Citus table distribution complete

[0;34m[INFO][0m Skipping 08-distribute-tables-rollback.sh (rollback script - run manually only)
[0;34m[INFO][0m Skipping 09-distribute-tables-test.sh (test script - set RUN_TESTS=true to enable)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 10-setup-cdc.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m CDC PIPELINE SETUP (Debezium + Elasticsearch Sink)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Log file: /var/log/fastorder/cdc/10-setup-cdc-*.log

[0;34m[INFO][0m Running CDC setup for identifier: coordinator
[2026-01-19 13:34:27] ==========================================
[2026-01-19 13:34:27] CDC SETUP SCRIPT STARTED
[2026-01-19 13:34:27] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260119_133427.log
[2026-01-19 13:34:27] ==========================================
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-19 13:34:28] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:34:28]   CDC Pipeline Setup (Debezium + ES Sink)
[2026-01-19 13:34:28] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:34:28]   Environment: user-sau-main-dev
[2026-01-19 13:34:28]   Identifier:  coordinator
[2026-01-19 13:34:28]   Service:     user
[2026-01-19 13:34:28] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:34:28] 📂 CDC_BASE_DIR exists: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc
[2026-01-19 13:34:28] Looking for service folder: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user
[2026-01-19 13:34:28] 
[2026-01-19 13:34:28] 📂 Found CDC configuration for service: user
[2026-01-19 13:34:28] Scanning for subservice directories in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user
[2026-01-19 13:34:28] Found subservice: contracts, checking for steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps
[2026-01-19 13:34:28] 
[2026-01-19 13:34:28] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:34:28]   Setting up CDC for: user/contracts
[2026-01-19 13:34:28] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:34:28] Found 8 step script(s) in /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps
[2026-01-19 13:34:28] 
[2026-01-19 13:34:28] 🔧 Running: 00-create-eav-tables.sh
[2026-01-19 13:34:28]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/00-create-eav-tables.sh
[2026-01-19 13:34:28]    Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Create EAV Tables for CDC Pipeline
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Identifier:  coordinator
  Tables:      user.contracts_int, user.contracts_json
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📝 Step 1: Creating EAV tables...
📥 Executing SQL...
  BEGIN
  CREATE TABLE
  CREATE INDEX
  CREATE INDEX
  CREATE INDEX
  CREATE FUNCTION
  psql:/tmp/create-eav-tables-user-sau-main-dev.sql:61: NOTICE:  trigger "trg_contracts_int_updated_at" for relation "user.contracts_int" does not exist, skipping
  DROP TRIGGER
  CREATE TRIGGER
  ALTER TABLE
  CREATE TABLE
  CREATE INDEX
  CREATE INDEX
  CREATE INDEX
  CREATE INDEX
  CREATE FUNCTION
  psql:/tmp/create-eav-tables-user-sau-main-dev.sql:120: NOTICE:  trigger "trg_contracts_json_updated_at" for relation "user.contracts_json" does not exist, skipping
  DROP TRIGGER
  CREATE TRIGGER
  ALTER TABLE
  COMMIT
  psql:/tmp/create-eav-tables-user-sau-main-dev.sql:161: NOTICE:  Created publication with all tables
  DO
  pubname         | schemaname |   tablename
  ------------------------+------------+----------------
  cdc_pub_user_contracts | user       | contracts
  cdc_pub_user_contracts | user       | contracts_int
  cdc_pub_user_contracts | user       | contracts_json
  (3 rows)
  
✅ EAV tables created

🔍 Step 2: Verifying tables...

📊 Table: user.contracts_int
                                Table "user.contracts_int"
   Column    |           Type           | Collation | Nullable |         Default          
-------------+--------------------------+-----------+----------+--------------------------
 id          | uuid                     |           | not null | utils.uuid_generate_v7()
 tenant_id   | character varying(100)   |           | not null | 
 contract_id | character(36)            |           | not null | 
 key         | character varying(100)   |           | not null | 
 value_int   | integer                  |           | not null | 
 created_at  | timestamp with time zone |           | not null | now()
 updated_at  | timestamp with time zone |           | not null | now()
Indexes:
    "contracts_int_pkey" PRIMARY KEY, btree (id)
    "idx_contracts_int_contract_id" btree (contract_id)
    "idx_contracts_int_key" btree (key)
    "idx_contracts_int_tenant_contract" btree (tenant_id, contract_id)
    "uq_contracts_int_contract_key" UNIQUE CONSTRAINT, btree (contract_id, key)
Foreign-key constraints:
    "fk_contracts_int_contract" FOREIGN KEY (contract_id) REFERENCES "user".contracts(id) ON DELETE CASCADE
Publications:
    "cdc_pub_user_contracts"

📊 Table: user.contracts_json
                               Table "user.contracts_json"
   Column    |           Type           | Collation | Nullable |         Default          
-------------+--------------------------+-----------+----------+--------------------------
 id          | uuid                     |           | not null | utils.uuid_generate_v7()
 tenant_id   | character varying(100)   |           | not null | 
 contract_id | character(36)            |           | not null | 
 key         | character varying(100)   |           | not null | 
 value_json  | jsonb                    |           | not null | 
 created_at  | timestamp with time zone |           | not null | now()
 updated_at  | timestamp with time zone |           | not null | now()
Indexes:
    "contracts_json_pkey" PRIMARY KEY, btree (id)
    "idx_contracts_json_contract_id" btree (contract_id)
    "idx_contracts_json_key" btree (key)
    "idx_contracts_json_tenant_contract" btree (tenant_id, contract_id)
    "idx_contracts_json_value_gin" gin (value_json)
    "uq_contracts_json_contract_key" UNIQUE CONSTRAINT, btree (contract_id, key)
Foreign-key constraints:
    "fk_contracts_json_contract" FOREIGN KEY (contract_id) REFERENCES "user".contracts(id) ON DELETE CASCADE
Publications:

📊 Publication Tables:
user.contracts
user.contracts_int
user.contracts_json

📝 Step 3: Sample data commands (for testing)...

-- Insert sample INT attributes (tenant_id must match parent contract)
INSERT INTO "user".contracts_int (tenant_id, contract_id, "key", value_int)
VALUES
    ('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'max_users', 100),
    ('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'credit_limit', 50000),
    ('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'tier_level', 2)
ON CONFLICT (contract_id, "key")
DO UPDATE SET value_int = EXCLUDED.value_int, updated_at = NOW();

-- Insert sample JSON attributes (tenant_id must match parent contract)
INSERT INTO "user".contracts_json (tenant_id, contract_id, "key", value_json)
VALUES
    ('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'metadata', '{"lang":"en","tier":"gold"}'::jsonb),
    ('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'settings', '{"notifications":true,"theme":"dark"}'::jsonb),
    ('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'permissions', '{"admin":true,"export":true}'::jsonb)
ON CONFLICT (contract_id, "key")
DO UPDATE SET value_json = EXCLUDED.value_json, updated_at = NOW();


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  EAV Tables Created Successfully
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Tables:
    - user.contracts_int
    - user.contracts_json

  Publication: cdc_pub_user_contracts

  Next Steps:
    1. Update Debezium connector table.include.list
    2. Setup ksqlDB pipeline (05-setup-ksqldb-pipeline.sh)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:34:29] ✅ Completed: 00-create-eav-tables.sh
[2026-01-19 13:34:29] 
[2026-01-19 13:34:29] 🔧 Running: 00b-migrate-tenant-id.sh
[2026-01-19 13:34:29]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/00b-migrate-tenant-id.sh
[2026-01-19 13:34:29]    Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Migration: Add tenant_id to EAV Tables
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Identifier:  coordinator
  Tables:      user.contracts_int, user.contracts_json
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔍 Pre-flight: Checking current state...
✅ tenant_id column already exists in both tables
✅ tenant_id is already NOT NULL - migration complete
[2026-01-19 13:34:30] ✅ Completed: 00b-migrate-tenant-id.sh
[2026-01-19 13:34:30] 
[2026-01-19 13:34:30] 🔧 Running: 01-setup-debezium-user-contracts.sh
[2026-01-19 13:34:30]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/01-setup-debezium-user-contracts.sh
[2026-01-19 13:34:30]    Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Debezium CDC Setup (User Contracts)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Identifier:  coordinator
  Table:       user.contracts
  Privacy:     Minimal user index (GDPR compliant)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔍 Verifying Kafka infrastructure...
✅ db-user-sau-main-dev-postgresql.fastorder.com resolves to 10.100.1.231
🔐 psql will use client cert for mTLS.
🔐 Retrieving credentials from secrets vault...
✅ Credentials retrieved from secrets vault
🔐 Writing Debezium credentials to FileConfigProvider secrets file...
[2026-01-19 13:34:33 UTC] USER=www-data EUID=0 PID=1350156 ACTION=passthru ARGS=sed -i s|^debezium.database.password=.*|debezium.database.password=EO6dTiheLAVHHEGTnCq6SoDuZ| /opt/kafka/secrets/user-sau-main-dev/coordinator/connector-secrets.properties
✅ Updated Debezium credentials in /opt/kafka/secrets/user-sau-main-dev/coordinator/connector-secrets.properties
🔐 Syncing debezium_user password in PostgreSQL...
✅ debezium_user password synchronized
🔍 Checking PostgreSQL SSL status...
✅ Server SSL is ON.
🔧 Applying schema, publication & grants over TLS…
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

DROP PUBLICATION
CREATE PUBLICATION
NOTICE:  Added user.contracts_int to publication
NOTICE:  Added user.contracts_json to publication
DO
GRANT
GRANT
GRANT
GRANT
GRANT
✅ Publication & grants done.
⏳ Waiting for Kafka Connect @ https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083/connectors…
[2026-01-19 13:34:34] 🔗 Waiting for Kafka Connect at: https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083
[2026-01-19 13:34:34] ⏳ Waiting for HTTP endpoint: https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083
[2026-01-19 13:34:34]    Expected codes: 200,500, timeout: 300s
[2026-01-19 13:34:34] ✅ HTTP endpoint ready: https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083 (code: 200, took: 0s)
[2026-01-19 13:34:34] 🔄 Testing Connect worker readiness...
[2026-01-19 13:34:34] ✅ Kafka Connect worker ready
🧹 Cleaning up existing Debezium connector and slot (if any)...
   Step 0a: Also resetting ES Sink connector offsets (required for coordinated reset)...
   → Deleting ES Sink connector offsets...
   → Creating temporary ES Sink placeholder for offset deletion...
{"error_code":400,"message":"Connector configuration is invalid and contains the following 2 error(s):\nCould not connect to Elasticsearch. Error message: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused\nFailed to create client to verify connection. java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused\nYou can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`"}{"error_code":404,"message":"Unknown connector pg_user_sau_main_dev_coordinator_user_contracts_es_sink"}   ⚠️  ES Sink offset deletion returned HTTP 404 (may be OK if no offsets existed)
   → Deleting ES Sink connector...
{"error_code":404,"message":"Connector pg_user_sau_main_dev_coordinator_user_contracts_es_sink not found"}   ✓ ES Sink connector cleanup complete
   Step 0b: Clearing stale Debezium connector offsets from Kafka Connect...
   → Stopping connector pg_user_sau_main_dev_user_contracts_debezium...
   → Deleting connector offsets (forces fresh snapshot)...
   ✓ Connector offsets deleted successfully (HTTP 200)
   Step 1: Deleting Debezium connector...
   Deleting connector: pg_user_sau_main_dev_user_contracts_debezium (attempt 1/10)
   ✓ Connector pg_user_sau_main_dev_user_contracts_debezium confirmed deleted
   Step 2: Waiting for replication slot to become inactive...
   ✓ Slot slot_user_sau_main_dev_user_contracts does not exist (clean state)
   Step 3: Dropping replication slot...
   ✓ Slot slot_user_sau_main_dev_user_contracts already dropped
   Step 4: Final verification...
✅ Cleanup complete - environment is clean for fresh CDC snapshot
🔐 Checking Debezium SSL certificate permissions...
🔍 Validating Debezium SSL certificates...
🔐 Connector will use mTLS to Postgres.
  ✓ Certificate: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt
  ✓ Key: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der
  ✓ Root CA: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
📤 Upserting connector: PUT https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_user_sau_main_dev_user_contracts_debezium/config
   Attempt 1/5: Sending PUT request to Kafka Connect...
   (This may take up to 60s as Connect validates the configuration)
   ✅ Success (HTTP 201)

🌐 HTTP Response: 201
✅ Connector upserted.
🔄 Verifying connector task startup...
✅ Debezium connector task is RUNNING
ℹ️  Source table user.contracts has 0 rows.
ℹ️  Snapshot will be metadata-only; offsets may stay empty until first change.
⏳ Waiting for Debezium initial snapshot to complete...
   📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
   📊 Debezium snapshot status: unknown
   ⏳ Snapshot in progress... (0s elapsed)
   ⏳ Snapshot in progress... (5s elapsed)
   ⏳ Snapshot in progress... (10s elapsed)
   📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
   📊 Debezium snapshot status: unknown
   ⏳ Snapshot in progress... (15s elapsed)
   ⏳ Snapshot in progress... (20s elapsed)
   ⏳ Snapshot in progress... (25s elapsed)
   📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
   📊 Debezium snapshot status: unknown
   ⏳ Snapshot in progress... (30s elapsed)
   ⏳ Snapshot in progress... (35s elapsed)
   ⏳ Snapshot in progress... (40s elapsed)
   📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
   📊 Debezium snapshot status: unknown
   ⏳ Snapshot in progress... (45s elapsed)
   ⏳ Snapshot in progress... (50s elapsed)
   ⏳ Snapshot in progress... (55s elapsed)
   📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
   📊 Debezium snapshot status: unknown
   ⏳ Snapshot in progress... (60s elapsed)
   ⏳ Snapshot in progress... (65s elapsed)
   ⏳ Snapshot in progress... (70s elapsed)
   📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
   📊 Debezium snapshot status: unknown
   ⏳ Snapshot in progress... (75s elapsed)
   ⏳ Snapshot in progress... (80s elapsed)
   ⏳ Snapshot in progress... (85s elapsed)
   📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
   📊 Debezium snapshot status: unknown
   ⏳ Snapshot in progress... (90s elapsed)
   ⏳ Snapshot in progress... (95s elapsed)
   ⏳ Snapshot in progress... (100s elapsed)
   📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
   📊 Debezium snapshot status: unknown
   ⏳ Snapshot in progress... (105s elapsed)
   ⏳ Snapshot in progress... (110s elapsed)
   ⏳ Snapshot in progress... (115s elapsed)

⚠️  Snapshot wait timeout (120s) on EMPTY table.
   Offsets are still empty, but source table has 0 rows.
   Proceeding anyway – CDC health will be verified by test inserts.
✅ Debezium connector is RUNNING after snapshot
🔍 Final verification: Checking Debezium offsets are recorded...
   ℹ️  Source table has 0 rows - skipping offset verification
✅ Debezium connector verified RUNNING (empty source table)
🔄 Phase 2: Updating connector to snapshot.mode=initial...
✅ Connector updated to snapshot.mode=initial (HTTP 200)
✅ Connector verified RUNNING after Phase 2 update
✅ Debezium connector configured successfully (two-phase snapshot complete)

==================================================================
MULTI-TABLE CDC Pipeline Configuration
==================================================================
   Tables:
     - user.contracts (main table)
     - user.contracts_int (EAV integer attributes)
     - user.contracts_json (EAV JSON attributes)

   Topics:
     - cdc.user.contracts
     - cdc.user.contracts_int
     - cdc.user.contracts_json

   COLUMN EXCLUSION (raw PII never leaves PostgreSQL):
     user.contracts.email,user.contracts.phone

   CAPTURED (safe for Kafka/ES):
     id (PK), tenant_id, home_region, username,
     display_name, email_hash, phone_hash, country_code,
     region_code, tags, segments, contract info

   DATA FLOW (Multi-Table CDC with ksqlDB Join):
     PostgreSQL Tables (1:N)
         ↓ Debezium (CDC per table)
         ↓ Kafka Topics (3 separate topics)
         ↓ ksqlDB (pivot + join → flat document)
         ↓ Compacted Topic (search.user.contracts.v1)
         ↓ ES Sink (UPSERT)
         ↓ Elasticsearch (flat search index)

   NEXT STEPS:
     1. Run 00-create-eav-tables.sh (if not done)
     2. Run 05-setup-ksqldb-pipeline.sh
     3. Run 06-setup-es-sink-ksqldb.sh
     4. Run 07-test-multi-table-cdc.sh
==================================================================
[2026-01-19 13:36:57] ✅ Completed: 01-setup-debezium-user-contracts.sh
[2026-01-19 13:36:57] 
[2026-01-19 13:36:57] 🔧 Running: 01b-install-ksqldb.sh
[2026-01-19 13:36:57]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/01b-install-ksqldb.sh
[2026-01-19 13:36:57]    Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ksqlDB Installation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Identifier:  coordinator
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  VM_IP:     10.100.1.234
  FQDN:      eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com

📦 Step 1: Checking Confluent Platform installation...
✅ ksqlDB already installed (version: )

📁 Step 2: Creating directories...
[2026-01-19 13:37:00 UTC] USER=www-data EUID=0 PID=1354536 ACTION=fsop ARGS=mkdir -p /var/lib/ksqldb/user-sau-main-dev/coordinator
[2026-01-19 13:37:00 UTC] USER=www-data EUID=0 PID=1354557 ACTION=fsop ARGS=mkdir -p /var/log/ksqldb/user-sau-main-dev/coordinator
[2026-01-19 13:37:00 UTC] USER=www-data EUID=0 PID=1354578 ACTION=fsop ARGS=mkdir -p /etc/ksqldb/user-sau-main-dev/coordinator
[2026-01-19 13:37:00 UTC] USER=www-data EUID=0 PID=1354599 ACTION=fsop ARGS=chown -R kafka:kafka /var/lib/ksqldb/user-sau-main-dev/coordinator /var/log/ksqldb/user-sau-main-dev/coordinator /etc/ksqldb/user-sau-main-dev/coordinator
✅ Directories created

⚙️  Step 3: Generating ksqlDB configuration...
[2026-01-19 13:37:00 UTC] USER=www-data EUID=0 PID=1354631 ACTION=fsop ARGS=mv /tmp/ksql-server-user-sau-main-dev.properties /etc/ksqldb/user-sau-main-dev/coordinator/ksql-server.properties
[2026-01-19 13:37:00 UTC] USER=www-data EUID=0 PID=1354655 ACTION=fsop ARGS=chown kafka:kafka /etc/ksqldb/user-sau-main-dev/coordinator/ksql-server.properties
[2026-01-19 13:37:00 UTC] USER=www-data EUID=0 PID=1354676 ACTION=fsop ARGS=chmod 640 /etc/ksqldb/user-sau-main-dev/coordinator/ksql-server.properties
✅ Configuration generated: /etc/ksqldb/user-sau-main-dev/coordinator/ksql-server.properties

🔧 Step 4: Creating systemd service...
[2026-01-19 13:37:00 UTC] USER=www-data EUID=0 PID=1354698 ACTION=fsop ARGS=mv /tmp/ksqldb-user-sau-main-dev-coordinator.service /etc/systemd/system/ksqldb-user-sau-main-dev-coordinator.service
[2026-01-19 13:37:00 UTC] USER=www-data EUID=0 PID=1354719 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-19 13:37:01 UTC] USER=www-data EUID=0 PID=1354803 ACTION=passthru ARGS=systemctl enable ksqldb-user-sau-main-dev-coordinator.service
✅ Systemd service created: ksqldb-user-sau-main-dev-coordinator.service

🚀 Step 5: Starting ksqlDB service...
🔍 Checking Kafka broker connectivity...
✅ Kafka broker is accessible
[2026-01-19 13:37:02 UTC] USER=www-data EUID=0 PID=1354878 ACTION=passthru ARGS=systemctl start ksqldb-user-sau-main-dev-coordinator.service
✅ ksqlDB service started
⏳ Waiting for ksqlDB to be ready...
✅ ksqlDB is ready!


🔍 Step 6: Verifying installation...

📊 Service Status:
[2026-01-19 13:37:02 UTC] USER=www-data EUID=0 PID=1354902 ACTION=passthru ARGS=systemctl status ksqldb-user-sau-main-dev-coordinator.service --no-pager -l
● ksqldb-user-sau-main-dev-coordinator.service - ksqlDB Server (user-sau-main-dev coordinator)
     Loaded: loaded (/etc/systemd/system/ksqldb-user-sau-main-dev-coordinator.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2026-01-19 13:19:34 UTC; 17min ago
       Docs: https://docs.ksqldb.io/
   Main PID: 1296228 (java)
      Tasks: 112 (limit: 19051)
     Memory: 514.2M
        CPU: 1min 37.926s
     CGroup: /system.slice/ksqldb-user-sau-main-dev-coordinator.service
             └─1296228 java -cp "/usr/share/java/ksqldb/*:/usr/share/java/rest-utils/*:/usr/share/java/confluent-common/*:" -Xms256m -Xmx512m -server -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:NewRatio=1 -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dksql.log.dir=/var/log/ksqldb/user-sau-main-dev/coordinator -Dlog4j.configuration=file:/etc/ksqldb/log4j.properties -Dksql.server.install.dir=/usr "-Xlog:gc*:file=/var/log/ksqldb/user-sau-main-dev/coordinator/ksql-server-gc.log:time,tags:filecount=10,filesize=102400" io.confluent.ksql.rest.server.KsqlServerMain /etc/ksqldb/user-sau-main-dev/coordinator/ksql-server.properties

Jan 19 13:36:43 web-03 ksql-server-start[1296228]: [2026-01-19 13:36:43,427] INFO Reporting thread saturation 0.001540631186208388 for _confluent-ksql-user-sau-main-dev_ksqldb_coordinatorquery_CTAS_CONTRACTS_JSON_AGG_289-f4d5c8dc-cd18-4200-9918-d2dee68acf57-StreamThread-2 (io.confluent.ksql.utilization.PersistentQuerySaturationMetrics:197)
Jan 19 13:36:43 web-03 ksql-server-start[1296228]: [2026-01-19 13:36:43,427] INFO Reporting thread saturation 0.0018373741570084463 for _confluent-ksql-user-sau-main-dev_ksqldb_coordinatorquery_CTAS_CONTRACTS_JSON_AGG_289-f4d5c8dc-cd18-4200-9918-d2dee68acf57-StreamThread-4 (io.confluent.ksql.utilization.PersistentQuerySaturationMetrics:197)
Jan 19 13:36:43 web-03 ksql-server-start[1296228]: [2026-01-19 13:36:43,427] INFO Reporting query saturation 0.0018373741570084463 for CTAS_CONTRACTS_JSON_AGG_289 (io.confluent.ksql.utilization.PersistentQuerySaturationMetrics:214)

📊 ksqlDB Info:
{
  "KsqlServerInfo": {
    "version": "7.6.5",
    "kafkaClusterId": "[2026-01-15 17:36:55 UTC] USER=www-data EUID=0 PID=455661 ACTION=passthru ARGS=bash -c cat /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/kafka_kraft_cluster_id\nvGsJvzNtQGKG1HQPRIaTPQ",
    "ksqlServiceId": "user-sau-main-dev_ksqldb_coordinator",
    "serverStatus": "RUNNING"
  }
}
✅ ksqlDB is responding

📡 Step 7: Registering ksqlDB to Observability API...
🔄 Registering ksqlDB node to observability dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       ksqlDB
[INFO]   Identifier:        user-sau-main-dev-ksqldb-coordinator
[INFO]   Identifier Parent: eventbus
[INFO]   IP:                10.100.1.234
[INFO]   Port:              8088
[INFO]   FQDN:              eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 07aaaced-f263-402d-90c8-50c9a9c0ff5c
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
✅ ksqlDB registered successfully

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ksqlDB Installation Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Service:  ksqldb-user-sau-main-dev-coordinator
  VM_IP:    10.100.1.234
  FQDN:     eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com
  Port:     8088
  Config:   /etc/ksqldb/user-sau-main-dev/coordinator/ksql-server.properties
  Data:     /var/lib/ksqldb/user-sau-main-dev/coordinator
  Logs:     /var/log/ksqldb/user-sau-main-dev/coordinator

  Dashboard:
    https://skeleton.dev.fastorder.com/dashboard/monitoring/environment2/<env-id>/service/ksqldb

  CLI Access (with SSL):
    ksql --ssl https://eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com:8088

  REST API (HTTPS):
    curl -k https://eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com:8088/info
    curl -k https://eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com:8088/ksql -H 'Content-Type: application/vnd.ksql.v1+json' -d '{"ksql": "SHOW STREAMS;"}'
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:37:02] ✅ Completed: 01b-install-ksqldb.sh
[2026-01-19 13:37:02] 
[2026-01-19 13:37:02] 🔧 Running: 02-setup-ksqldb-pipeline.sh
[2026-01-19 13:37:02]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/02-setup-ksqldb-pipeline.sh
[2026-01-19 13:37:02]    Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ksqlDB CDC Pipeline Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Tables:      user.contracts, contracts_int, contracts_json
  Output:      user_sau_main_dev_user_contracts
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔍 Step 0: Checking ksqlDB availability...
✅ ksqlDB is running (version: 7.6.5)

📦 Step 1: Creating compacted output topic...
📥 Creating compacted topic: user_sau_main_dev_user_contracts
[2026-01-19 13:37:05 UTC] USER=www-data EUID=0 PID=1355538 ACTION=passthru ARGS=sudo -u kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com:9092 --command-config /tmp/kafka-client-user-sau-main-dev.properties --create --topic user_sau_main_dev_user_contracts --partitions 12 --replication-factor 1 --config cleanup.policy=compact --config min.compaction.lag.ms=0 --config delete.retention.ms=86400000 --config segment.ms=3600000
Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:561)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:512)
	at org.apache.kafka.clients.admin.Admin.create(Admin.java:137)
	at org.apache.kafka.tools.TopicCommand$TopicService.createAdminClient(TopicCommand.java:456)
	at org.apache.kafka.tools.TopicCommand$TopicService.<init>(TopicCommand.java:445)
	at org.apache.kafka.tools.TopicCommand.execute(TopicCommand.java:101)
	at org.apache.kafka.tools.TopicCommand.mainNoExit(TopicCommand.java:90)
	at org.apache.kafka.tools.TopicCommand.main(TopicCommand.java:85)
Caused by: org.apache.kafka.common.KafkaException: Failed to create new NetworkClient
	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:255)
	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:190)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:545)
	... 7 more
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /opt/kafka/secrets/user-sau-main-dev/coordinator/kafka.client.keystore.p12 of type JKS
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:380)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:352)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:302)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:162)
	at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:147)
	at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:100)
	at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:70)
	at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:193)
	at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:82)
	at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:120)
	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:224)
	... 9 more
Caused by: java.io.IOException: keystore password was incorrect
	at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159)
	at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:242)
	at java.base/java.security.KeyStore.load(KeyStore.java:1473)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:377)
	... 19 more
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
	... 23 more

✅ Topic created with compaction enabled

🧹 Step 1b: Checking for existing ksqlDB objects to clean up...
✅ No existing objects found - proceeding with fresh creation

📝 Step 2: Generating ksqlDB DDL...
✅ DDL generated: /tmp/ksql-user-contracts-user-sau-main-dev.ksql

🚀 Step 3: Executing ksqlDB DDL...
📋 Executing DDL statements...
  → CREATE STREAM IF NOT EXISTS contracts_stream (...
    ⚠️  
  → CREATE STREAM IF NOT EXISTS contracts_int_stream (...
    ⚠️  
  → CREATE STREAM IF NOT EXISTS contracts_json_stream (...
    ⚠️  
  → CREATE TABLE IF NOT EXISTS contracts_int_agg...
    ⚠️  
  → SELECT...
    ⚠️  
  → CREATE TABLE IF NOT EXISTS contracts_json_agg...
    ⚠️  
  → SELECT...
    ⚠️  
  → CREATE TABLE IF NOT EXISTS contracts_tbl...
    ⚠️  
  → SELECT...
    ⚠️  
  → CREATE TABLE IF NOT EXISTS user_search_doc_raw...
    ⚠️  
  → SELECT...
    ⚠️  
  → CREATE STREAM IF NOT EXISTS user_search_doc_keyed...
    ⚠️  
  → SELECT...
    ⚠️  
  → PARTITION BY `doc_id`...
    ⚠️  

🔍 Step 4: Verifying ksqlDB objects...

📊 Streams:

📊 Tables:

📊 Running Queries:
  Active queries: 

💾 Step 5: Saving DDL for reference...
[2026-01-19 13:37:08 UTC] USER=www-data EUID=0 PID=1356244 ACTION=passthru ARGS=mkdir -p /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/ksqldb
[2026-01-19 13:37:08 UTC] USER=www-data EUID=0 PID=1356265 ACTION=passthru ARGS=cp /tmp/ksql-user-contracts-user-sau-main-dev.ksql /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/ksqldb/user-contracts-pipeline.ksql
✅ DDL saved to: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/ksqldb/user-contracts-pipeline.ksql

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ksqlDB Pipeline Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Input Topics:
    - user_sau_main_dev_cdc.user.contracts
    - user_sau_main_dev_cdc.user.contracts_int
    - user_sau_main_dev_cdc.user.contracts_json

  Output Topic:
    - user_sau_main_dev_user_contracts (compacted)

  ksqlDB Objects:
    - Streams: contracts_stream, contracts_int_stream, contracts_json_stream
    - Tables: contracts_tbl (keyed by doc_id)
    - Tables: contracts_int_agg, contracts_json_agg (keyed by doc_id)
    - Tables: user_search_doc_raw (joined table)
    - Streams: user_search_doc_keyed (final output)

  Join Key: doc_id = CONCAT(tenant_id, ':', contract_id)

  Next Steps:
    1. Update ES Sink to consume from: user_sau_main_dev_user_contracts
    2. Test with data insertion
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-19 13:37:08] ✅ Completed: 02-setup-ksqldb-pipeline.sh
[2026-01-19 13:37:08] 
[2026-01-19 13:37:08] 🔧 Running: 03-setup-es-sink-ksqldb.sh
[2026-01-19 13:37:08]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/03-setup-es-sink-ksqldb.sh
[2026-01-19 13:37:08]    Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ES Sink Connector (ksqlDB Joined Topic)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: user-sau-main-dev
  Identifier:  coordinator
  Input:       user_sau_main_dev_user_contracts
  Output:      user_sau_main_dev_user_contracts (index)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔐 Step 1: Getting Elasticsearch credentials...
✅ Elasticsearch credentials loaded
✅ SSL passwords loaded

🔒 Step 1c: Ensuring ES client keystore and truststore are properly configured...
📦 Creating ES client keystore from PEM certificates...
[2026-01-19 13:37:12 UTC] USER=www-data EUID=0 PID=1356439 ACTION=passthru ARGS=openssl pkcs12 -export -in /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.crt -inkey /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.key -certfile /etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt -name es-client -out /opt/kafka/secrets/user-sau-main-dev/coordinator/es-client.keystore.p12 -password pass:OV9hCGeLdjgcwFFaqhyU34SjH3OUk4uu
ERROR: passthru not allowed: openssl
[2026-01-19 13:37:12] ❌ FAILED: 03-setup-es-sink-ksqldb.sh (exit code: 1)
[2026-01-19 13:37:12] ❌ CRITICAL: This is a required step for CDC pipeline. Aborting.

[0;31m[ERROR][0m ❌ Database infrastructure (postgresql) setup failed with exit code: 1

📥 Download Full Logs

06-finalizing local

⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

📄 View Logs (0 chars)

Loading logs...

Total Steps

Succeeded

Failed

Running

Pending

10 minutes

Total Steps Time

← Back to Dashboard 🔍 View Environment