📊 Provisioning Job Status

[INFO] Using eventbus engine from EVENTBUS_ENGINE environment variable: kafka
[INFO] Cleaning up any existing locks...

Starting eventbus engine: kafka
═══════════════════════════════════════════════

[INFO] Using environment from web interface: zone-universe-main-dev
[2026-02-05 10:25:18] Using web-provided environment: zone-universe-main-dev
[2026-02-05 10:25:18] Service: zone, Zone: universe, Branch: main, Env: dev
✓ Environment initialized successfully (mode: general)
[INFO] Starting Kafka setup process...
[INFO] Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps
[INFO] Environment: zone-universe-main-dev

[INFO] Found 13 step(s) to execute

[INFO] 📦 Step 1/13: install debezium connector...
═══════════════════════════════════════════════════════════════════
Fetching latest versions from Maven Central...
Installing Debezium PostgreSQL Connector
  Debezium version: 3.4.1.Final
  pgjdbc version:   42.7.9
═══════════════════════════════════════════════════════════════════
[OK] Debezium 3.4.1.Final with pgjdbc 42.7.9 already installed
[OK] ✅ Step 1 completed: 00-install-debezium-connector.sh

[INFO] 📦 Step 2/13: kafka setup...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials for secrets vault...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
🧹 Checking for orphaned Kafka processes on ports 9092, 9093, 8083...
  ⚠️  Found process on port 9092 (PIDs: [2026-02-05 10:25:19 UTC] USER=www-data EUID=0 PID=750678 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 2>/dev/null || true
638366
638367
638454
638579
638833
638834
646982
647929
716246), killing...
[2026-02-05 10:25:19 UTC] USER=www-data EUID=0 PID=750689 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 10:25:19 UTC] USER=www-data EUID=0 PID=750678 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 2>/dev/null || true
638366
638367
638454
638579
638833
638834
646982
647929
716246 2>/dev/null || true
/usr/bin/bash: line 2: 638366: command not found
/usr/bin/bash: line 3: 638367: command not found
/usr/bin/bash: line 4: 638454: command not found
/usr/bin/bash: line 5: 638579: command not found
/usr/bin/bash: line 6: 638833: command not found
/usr/bin/bash: line 7: 638834: command not found
/usr/bin/bash: line 8: 646982: command not found
/usr/bin/bash: line 9: 647929: command not found
  ⚠️  Found process on port 9093 (PIDs: [2026-02-05 10:25:20 UTC] USER=www-data EUID=0 PID=750724 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 2>/dev/null || true
637045
637832
637835
638366
638454
638833
641153
652982
686541
716246), killing...
[2026-02-05 10:25:21 UTC] USER=www-data EUID=0 PID=750756 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 10:25:20 UTC] USER=www-data EUID=0 PID=750724 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 2>/dev/null || true
637045
637832
637835
638366
638454
638833
641153
652982
686541
716246 2>/dev/null || true
/usr/bin/bash: line 2: 637045: command not found
/usr/bin/bash: line 3: 637832: command not found
/usr/bin/bash: line 4: 637835: command not found
/usr/bin/bash: line 5: 638366: command not found
/usr/bin/bash: line 6: 638454: command not found
/usr/bin/bash: line 7: 638833: command not found
/usr/bin/bash: line 8: 641153: command not found
/usr/bin/bash: line 9: 652982: command not found
/usr/bin/bash: line 10: 686541: command not found
  ⚠️  Found process on port 8083 (PIDs: [2026-02-05 10:25:22 UTC] USER=www-data EUID=0 PID=750800 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 2>/dev/null || true
638367
638579
638834), killing...
[2026-02-05 10:25:22 UTC] USER=www-data EUID=0 PID=750823 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 10:25:22 UTC] USER=www-data EUID=0 PID=750800 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 2>/dev/null || true
638367
638579
638834 2>/dev/null || true
/usr/bin/bash: line 2: 638367: command not found
/usr/bin/bash: line 3: 638579: command not found
✅ Port cleanup completed
Ensuring KAFKA application environment for coordinator...
[INFO] Using existing KAFKA environment: eventbus-zone-universe-main-dev-kafka-connect.fastorder.com (10.100.1.110)
Ensuring KAFKA_BROKER_IP application environment for coordinator...
[INFO] Using existing KAFKA BROKER environment: eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com (10.100.1.52)
[INFO] Kafka Broker IP: 10.100.1.52
[INFO] Kafka Connect IP: 10.100.1.110
[INFO] Registered /etc/hosts: eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com -> 10.100.1.52
[INFO] Registered /etc/hosts: eventbus-zone-universe-main-dev-kafka-connect.fastorder.com -> 10.100.1.110
🔐 Initializing keystore passwords...
[INFO] 🔍 Checking secrets backend (provider: aws)...
✅ Retrieved passwords from remote backend
[INFO] ✅ Using existing passwords from backend
✅ Keystore passwords initialized
   - Keystore password: sfDzrJOG... (32 chars)
   - Truststore password: JBH3gyVt... (32 chars)
[INFO] 🔐 Vaulting kafka passwords to remote backend...
✅ Passwords vaulted to remote backend
✅ Kafka keystore passwords saved to AWS Secrets Manager
[INFO] Generating for: zone-universe-main-dev (host=eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com ip=10.100.1.52)
[2026-02-05 10:25:33 UTC] USER=www-data EUID=0 PID=751387 ACTION=fsop ARGS=rm -rf /opt/kafka/secrets/zone-universe-main-dev/coordinator /data/kafka/zone-universe-main-dev/coordinator
[2026-02-05 10:25:33 UTC] USER=www-data EUID=0 PID=751396 ACTION=fsop ARGS=mkdir -p /opt/kafka/secrets/zone-universe-main-dev/coordinator /opt/kafka/config/zone-universe-main-dev/coordinator /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem /data/kafka/zone-universe-main-dev_coordinator-data
[2026-02-05 10:25:33 UTC] USER=www-data EUID=0 PID=751405 ACTION=fsop ARGS=chown -R kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator
[2026-02-05 10:25:34 UTC] USER=www-data EUID=0 PID=751418 ACTION=fsop ARGS=chown -R kafka:kafka /opt/kafka/config/zone-universe-main-dev/coordinator /data/kafka/zone-universe-main-dev_coordinator-data
[2026-02-05 10:25:34 UTC] USER=www-data EUID=0 PID=751455 ACTION=fsop ARGS=chmod 750 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem
[2026-02-05 10:25:34 UTC] USER=www-data EUID=0 PID=751466 ACTION=fsop ARGS=chmod 700 /tmp/fo-tls.vwUuhI
[2026-02-05 10:25:34 UTC] USER=www-data EUID=0 PID=751475 ACTION=fsop ARGS=chmod 755 /tmp/fo-tls.vwUuhI
[2026-02-05 10:25:34 UTC] USER=www-data EUID=0 PID=751484 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/fo-tls.vwUuhI/ra_root.crt
[2026-02-05 10:25:34 UTC] USER=www-data EUID=0 PID=751493 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/fo-tls.vwUuhI/ra_root.key
[2026-02-05 10:25:34 UTC] USER=www-data EUID=0 PID=751502 ACTION=fsop ARGS=chmod 644 /tmp/fo-tls.vwUuhI/ra_root.crt
[2026-02-05 10:25:34 UTC] USER=www-data EUID=0 PID=751511 ACTION=fsop ARGS=chmod 644 /tmp/fo-tls.vwUuhI/ra_root.key
Certificate was added to keystore
[2026-02-05 10:25:36 UTC] USER=www-data EUID=0 PID=751583 ACTION=fsop ARGS=mv /tmp/fo-tls.vwUuhI/truststore.jks /opt/kafka/secrets/zone-universe-main-dev/coordinator/truststore.jks
[2026-02-05 10:25:36 UTC] USER=www-data EUID=0 PID=751593 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/zone-universe-main-dev/coordinator/truststore.jks
[2026-02-05 10:25:36 UTC] USER=www-data EUID=0 PID=751606 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/truststore.jks
Generating 4,096 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 825 days
	for: CN=eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com, OU=Kafka Broker, O=FastOrder, C=AE

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /tmp/fo-tls.vwUuhI/kafka.server.keystore.jks -destkeystore /tmp/fo-tls.vwUuhI/kafka.server.keystore.jks -deststoretype pkcs12".

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /tmp/fo-tls.vwUuhI/kafka.server.keystore.jks -destkeystore /tmp/fo-tls.vwUuhI/kafka.server.keystore.jks -deststoretype pkcs12".
Certificate request self-signature ok
subject=C = AE, O = FastOrder, OU = Kafka Broker, CN = eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com
Certificate was added to keystore

Warning:
Certificate reply was installed in keystore

Warning:
[2026-02-05 10:25:41 UTC] USER=www-data EUID=0 PID=751818 ACTION=fsop ARGS=mv /tmp/fo-tls.vwUuhI/kafka.server.keystore.jks /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 10:25:41 UTC] USER=www-data EUID=0 PID=751839 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 10:25:41 UTC] USER=www-data EUID=0 PID=751848 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.server.keystore.jks
Generating 4,096 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 825 days
	for: CN=eventbus-zone-universe-main-dev-kafka-connect.fastorder.com, OU=Kafka Connect REST, O=FastOrder, C=AE
Certificate request self-signature ok
subject=C = AE, O = FastOrder, OU = Kafka Connect REST, CN = eventbus-zone-universe-main-dev-kafka-connect.fastorder.com
Certificate was added to keystore
Certificate reply was installed in keystore
[2026-02-05 10:25:52 UTC] USER=www-data EUID=0 PID=752279 ACTION=fsop ARGS=mv /tmp/fo-tls.vwUuhI/connect-rest.keystore.p12 /opt/kafka/secrets/zone-universe-main-dev/coordinator/connect-rest.keystore.p12
[2026-02-05 10:25:52 UTC] USER=www-data EUID=0 PID=752293 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/zone-universe-main-dev/coordinator/connect-rest.keystore.p12
[2026-02-05 10:25:52 UTC] USER=www-data EUID=0 PID=752302 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/connect-rest.keystore.p12
Certificate request self-signature ok
subject=CN = kafka-client-zone-universe-main-dev, OU = Kafka Client, O = FastOrder, C = AE
[2026-02-05 10:25:52 UTC] USER=www-data EUID=0 PID=752320 ACTION=fsop ARGS=cp /tmp/fo-tls.vwUuhI/ra_root.crt /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem
[2026-02-05 10:25:52 UTC] USER=www-data EUID=0 PID=752329 ACTION=fsop ARGS=cp /tmp/fo-tls.vwUuhI/client-key.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:25:52 UTC] USER=www-data EUID=0 PID=752338 ACTION=fsop ARGS=cp /tmp/fo-tls.vwUuhI/client-cert.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
[2026-02-05 10:25:52 UTC] USER=www-data EUID=0 PID=752347 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
[2026-02-05 10:25:52 UTC] USER=www-data EUID=0 PID=752356 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
[2026-02-05 10:25:52 UTC] USER=www-data EUID=0 PID=752367 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:25:52 UTC] USER=www-data EUID=0 PID=752378 ACTION=fsop ARGS=mv /tmp/fo-tls.vwUuhI/kafka.client.keystore.p12 /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.client.keystore.p12
[2026-02-05 10:25:52 UTC] USER=www-data EUID=0 PID=752393 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.client.keystore.p12
[2026-02-05 10:25:52 UTC] USER=www-data EUID=0 PID=752403 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.client.keystore.p12
🔐 Ensuring kafka user has access to PostgreSQL certificates...
✅ kafka is already in postgres group
🧹 Cleaning up conflicting services and processes on Kafka ports on 10.100.1.52...
🔪 Killing processes on 10.100.1.52:8083: [2026-02-05 10:25:53 UTC] USER=www-data EUID=0 PID=752449 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.52:8083 -t 2>/dev/null || true
[2026-02-05 10:25:53 UTC] USER=www-data EUID=0 PID=752470 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 10:25:53 UTC] USER=www-data EUID=0 PID=752449 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.52:8083 -t 2>/dev/null || true
🔪 Killing processes on 10.100.1.52:9092: [2026-02-05 10:25:53 UTC] USER=www-data EUID=0 PID=752481 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.52:9092 -t 2>/dev/null || true
716246
[2026-02-05 10:25:54 UTC] USER=www-data EUID=0 PID=752522 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 10:25:53 UTC] USER=www-data EUID=0 PID=752481 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.52:9092 -t 2>/dev/null || true
716246
/usr/bin/bash: line 2: 716246: command not found
🔪 Killing processes on 10.100.1.52:9093: [2026-02-05 10:25:54 UTC] USER=www-data EUID=0 PID=752533 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.52:9093 -t 2>/dev/null || true
716246
[2026-02-05 10:25:54 UTC] USER=www-data EUID=0 PID=752564 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 10:25:54 UTC] USER=www-data EUID=0 PID=752533 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.52:9093 -t 2>/dev/null || true
716246
/usr/bin/bash: line 2: 716246: command not found
✅ Port cleanup completed
🔧 Checking for Kafka Connect internal topics with incorrect cleanup policy...
📋 Kafka broker is running, checking topic cleanup policies...
✅ Topic cleanup policy fix completed
🔧 Creating environment-specific systemd units...
🔧 Writing client properties to /etc/kafka/client-zone-universe-main-dev-coordinator.properties ...
[2026-02-05 10:26:12 UTC] USER=www-data EUID=0 PID=754524 ACTION=fsop ARGS=chown root:kafka /etc/kafka/client-zone-universe-main-dev-coordinator.properties
[2026-02-05 10:26:12 UTC] USER=www-data EUID=0 PID=754540 ACTION=fsop ARGS=chmod 0644 /etc/kafka/client-zone-universe-main-dev-coordinator.properties
🔧 Creating PEM certificates for PHP mTLS access...
[2026-02-05 10:26:12 UTC] USER=www-data EUID=0 PID=754551 ACTION=passthru ARGS=bash -c openssl pkcs12 -in '/opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.client.keystore.p12' -clcerts -nokeys -passin pass:'sfDzrJOGShZ7iICxdMeMU72wUc1Es4tN' -out '/opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client.crt' 2>/dev/null
[2026-02-05 10:26:12 UTC] USER=www-data EUID=0 PID=754564 ACTION=passthru ARGS=bash -c openssl pkcs12 -in '/opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.client.keystore.p12' -nocerts -nodes -passin pass:'sfDzrJOGShZ7iICxdMeMU72wUc1Es4tN' -out '/opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client.key' 2>/dev/null
[2026-02-05 10:26:12 UTC] USER=www-data EUID=0 PID=754579 ACTION=passthru ARGS=bash -c keytool -exportcert -alias fastorder-ra-root -keystore '/opt/kafka/secrets/zone-universe-main-dev/coordinator/truststore.jks' -storepass 'JBH3gyVtYBjR06bIlHqf0792HMBfdcRK' -rfc -file '/opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.crt' 2>/dev/null
[2026-02-05 10:26:14 UTC] USER=www-data EUID=0 PID=754664 ACTION=fsop ARGS=chown root:kafka /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client.crt /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client.key /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.crt
[2026-02-05 10:26:14 UTC] USER=www-data EUID=0 PID=754673 ACTION=fsop ARGS=chmod 0644 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client.crt /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.crt
🔐 Creating connector secrets file for FileConfigProvider...
[2026-02-05 10:26:14 UTC] USER=www-data EUID=0 PID=754710 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/zone-universe-main-dev/coordinator/connector-secrets.properties
[2026-02-05 10:26:14 UTC] USER=www-data EUID=0 PID=754722 ACTION=fsop ARGS=chmod 0600 /opt/kafka/secrets/zone-universe-main-dev/coordinator/connector-secrets.properties
✅ Connector secrets file created: /opt/kafka/secrets/zone-universe-main-dev/coordinator/connector-secrets.properties
   FileConfigProvider syntax: ${file:/opt/kafka/secrets/zone-universe-main-dev/coordinator/connector-secrets.properties:key_name}
🔧 Creating Canary Event timer for pipeline verification...
[2026-02-05 10:26:14 UTC] USER=www-data EUID=0 PID=754749 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 10:26:15 UTC] USER=www-data EUID=0 PID=754802 ACTION=passthru ARGS=systemctl enable kafka-canary-zone-universe-main-dev.timer
[2026-02-05 10:26:16 UTC] USER=www-data EUID=0 PID=754862 ACTION=passthru ARGS=systemctl start kafka-canary-zone-universe-main-dev.timer
✅ Canary timer installed: kafka-canary-zone-universe-main-dev.timer (every 5 minutes)
[2026-02-05 10:26:16 UTC] USER=www-data EUID=0 PID=754872 ACTION=passthru ARGS=systemctl daemon-reload
Failed to print table: Broken pipe
Failed to print table: Broken pipe
🔒 Adjusting group ownership and permissions ...
[2026-02-05 10:26:24 UTC] USER=www-data EUID=0 PID=755211 ACTION=fsop ARGS=chown :kafka /opt/kafka/secrets/zone-universe-main-dev/coordinator/truststore.jks /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 10:26:25 UTC] USER=www-data EUID=0 PID=755230 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/truststore.jks /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 10:26:25 UTC] USER=www-data EUID=0 PID=755240 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.client.keystore.p12
[2026-02-05 10:26:25 UTC] USER=www-data EUID=0 PID=755249 ACTION=fsop ARGS=chown root:kafka /etc/kafka/client-zone-universe-main-dev-coordinator.properties
[2026-02-05 10:26:25 UTC] USER=www-data EUID=0 PID=755258 ACTION=fsop ARGS=chmod 0644 /etc/kafka/client-zone-universe-main-dev-coordinator.properties

✅ Kafka configuration complete for zone-universe-main-dev_coordinator
  Broker ID         : 57
  Broker keystore   : /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.server.keystore.jks
  REST keystore     : /opt/kafka/secrets/zone-universe-main-dev/coordinator/connect-rest.keystore.p12
  Truststore        : /opt/kafka/secrets/zone-universe-main-dev/coordinator/truststore.jks
  Client PKCS12     : /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.client.keystore.p12
  Data directory    : /data/kafka/zone-universe-main-dev_coordinator-data
  Server config     : /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
  Connect config    : /opt/kafka/config/zone-universe-main-dev/coordinator/connect-distributed.properties
  CLI client config : /etc/kafka/client-zone-universe-main-dev-coordinator.properties

🎯 Next step: Run 03-restart-kafka-related-services.sh to start services

[OK] ✅ Step 2 completed: 01-kafka-setup.sh

[INFO] 📦 Step 3/13: metadata...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] 🎯 Kafka metadata mode: kraft

╔════════════════════════════════════════════════════════════════════╗
║            Kafka Metadata Layer Setup                             ║
╚════════════════════════════════════════════════════════════════════╝

  Environment    : zone-universe-main-dev
  Service        : zone
  Zone           : universe
  Branch         : main
  Environment    : dev
  VM IP          : 142.93.238.16
  Metadata Mode  : kraft

  📋 KRaft Mode (Modern)
  ────────────────────────────────────────────────────────────────
  ✅ No ZooKeeper dependency
  ✅ Faster metadata operations
  ✅ Simplified architecture
  ✅ Recommended for new deployments
  ⚠️  Requires Kafka 3.3+ in production


════════════════════════════════════════════════════════════════════

[INFO] 🚀 Executing KRaft setup script...
[INFO] Script: /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps/metadata/kraft.sh

[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[2026-02-05 10:26:26 UTC] USER=www-data EUID=0 PID=755351 ACTION=fsop ARGS=mkdir -p /data/kafka/zone-universe-main-dev_coordinator-meta /opt/kafka/config/zone-universe-main-dev/coordinator /data/kafka/zone-universe-main-dev_coordinator-data
[2026-02-05 10:26:26 UTC] USER=www-data EUID=0 PID=755360 ACTION=fsop ARGS=chown -R kafka:kafka /data/kafka/zone-universe-main-dev_coordinator-meta /opt/kafka/config/zone-universe-main-dev/coordinator /data/kafka/zone-universe-main-dev_coordinator-data
[2026-02-05 10:26:26 UTC] USER=www-data EUID=0 PID=755370 ACTION=fsop ARGS=chmod 770 /data/kafka/zone-universe-main-dev_coordinator-meta /opt/kafka/config/zone-universe-main-dev/coordinator /data/kafka/zone-universe-main-dev_coordinator-data
[INFO] Adding eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com to /etc/hosts -> 10.100.1.52
[INFO] Adding eventbus-zone-universe-main-dev-kafka-connect.fastorder.com to /etc/hosts -> 10.100.1.110
[INFO] Setting up KRaft for: zone-universe-main-dev (host=eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com ip=10.100.1.52)
[2026-02-05 10:26:26 UTC] USER=www-data EUID=0 PID=755399 ACTION=fsop ARGS=mkdir -p /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev
[INFO] Using existing cluster.id from state

🔧 Configuring Kafka for KRaft mode...
[2026-02-05 10:26:27 UTC] USER=www-data EUID=0 PID=755422 ACTION=fsop ARGS=test -f /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 10:26:27 UTC] USER=www-data EUID=0 PID=755447 ACTION=fsop ARGS=sed -i /^zookeeper\.connect=/d /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 10:26:27 UTC] USER=www-data EUID=0 PID=755458 ACTION=passthru ARGS=bash -c grep -q '^process.roles=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 10:26:27 UTC] USER=www-data EUID=0 PID=755488 ACTION=passthru ARGS=bash -c grep -q '^node.id=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 10:26:27 UTC] USER=www-data EUID=0 PID=755507 ACTION=passthru ARGS=bash -c grep -q '^broker.id=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 10:26:27 UTC] USER=www-data EUID=0 PID=755527 ACTION=passthru ARGS=bash -c grep -q '^controller.listener.names=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 10:26:27 UTC] USER=www-data EUID=0 PID=755546 ACTION=passthru ARGS=bash -c grep -q '^controller.quorum.voters=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 10:26:27 UTC] USER=www-data EUID=0 PID=755568 ACTION=passthru ARGS=bash -c grep -q '^metadata.log.dir=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755599 ACTION=fsop ARGS=sed -i s|^log.dirs=.*|log.dirs=/data/kafka/zone-universe-main-dev_coordinator-data| /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755614 ACTION=passthru ARGS=bash -c grep -q '^listeners=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755623 ACTION=passthru ARGS=bash -c grep -q 'CONTROLLER://' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755632 ACTION=fsop ARGS=sed -i s|^listeners=.*|listeners=SSL://10.100.1.52:9092,CONTROLLER://10.100.1.52:9093| /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755642 ACTION=passthru ARGS=bash -c grep -q '^advertised.listeners=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755651 ACTION=fsop ARGS=sed -i s|^advertised.listeners=.*|advertised.listeners=SSL://eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com:9092| /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755660 ACTION=passthru ARGS=bash -c grep -q '^listener.security.protocol.map=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755669 ACTION=fsop ARGS=sed -i s|^listener.security.protocol.map=.*|listener.security.protocol.map=SSL:SSL,CONTROLLER:PLAINTEXT| /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755679 ACTION=passthru ARGS=bash -c grep -q '^inter.broker.listener.name=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755688 ACTION=fsop ARGS=sed -i s|^inter.broker.listener.name=.*|inter.broker.listener.name=SSL| /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755699 ACTION=passthru ARGS=bash -c grep -q '^offsets.topic.replication.factor=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755708 ACTION=fsop ARGS=sed -i s|^offsets.topic.replication.factor=.*|offsets.topic.replication.factor=1| /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755717 ACTION=passthru ARGS=bash -c grep -q '^transaction.state.log.replication.factor=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755726 ACTION=fsop ARGS=sed -i s|^transaction.state.log.replication.factor=.*|transaction.state.log.replication.factor=1| /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755738 ACTION=passthru ARGS=bash -c grep -q '^transaction.state.log.min.isr=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755748 ACTION=fsop ARGS=sed -i s|^transaction.state.log.min.isr=.*|transaction.state.log.min.isr=1| /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755757 ACTION=passthru ARGS=bash -c grep -q '^min.insync.replicas=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
  ✅ KRaft configuration applied to server.properties
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755776 ACTION=fsop ARGS=test -f /data/kafka/zone-universe-main-dev_coordinator-meta/meta.properties
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755785 ACTION=fsop ARGS=test -f /data/kafka/zone-universe-main-dev_coordinator-data/meta.properties
[INFO] Already formatted: both /data/kafka/zone-universe-main-dev_coordinator-meta and /data/kafka/zone-universe-main-dev_coordinator-data have meta.properties
🔧 Creating/refreshing KRaft systemd unit...
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755803 ACTION=fsop ARGS=sed -i s|\\$MAINPID|$MAINPID|g /etc/systemd/system/confluent-kraft-zone-universe-main-dev_coordinator.service
[2026-02-05 10:26:28 UTC] USER=www-data EUID=0 PID=755812 ACTION=passthru ARGS=systemctl daemon-reload
  ✅ Ensured confluent-kraft-zone-universe-main-dev_coordinator.service
🛑 Stopping legacy ZooKeeper-mode services and current KRaft instance...
  🛑 Stopping current: confluent-kraft-zone-universe-main-dev_coordinator.service
[2026-02-05 10:26:29 UTC] USER=www-data EUID=0 PID=755864 ACTION=passthru ARGS=systemctl stop confluent-kraft-zone-universe-main-dev_coordinator.service
  🧹 Cleaning up rogue Kafka processes...
  🧹 Killing any processes holding Kafka ports 9092, 9093...
  🔪 Killing processes on port 9092: 638366
638367
638454
638579
638833
638834
646982
647929
[2026-02-05 10:26:33 UTC] USER=www-data EUID=0 PID=756118 ACTION=passthru ARGS=bash -c kill -9 638366
[2026-02-05 10:26:33 UTC] USER=www-data EUID=0 PID=756128 ACTION=passthru ARGS=bash -c kill -9 638367
[2026-02-05 10:26:33 UTC] USER=www-data EUID=0 PID=756137 ACTION=passthru ARGS=bash -c kill -9 638454
[2026-02-05 10:26:33 UTC] USER=www-data EUID=0 PID=756146 ACTION=passthru ARGS=bash -c kill -9 638579
[2026-02-05 10:26:33 UTC] USER=www-data EUID=0 PID=756157 ACTION=passthru ARGS=bash -c kill -9 638833
[2026-02-05 10:26:33 UTC] USER=www-data EUID=0 PID=756169 ACTION=passthru ARGS=bash -c kill -9 638834
[2026-02-05 10:26:33 UTC] USER=www-data EUID=0 PID=756179 ACTION=passthru ARGS=bash -c kill -9 646982
[2026-02-05 10:26:33 UTC] USER=www-data EUID=0 PID=756188 ACTION=passthru ARGS=bash -c kill -9 647929
  🔪 Killing processes on port 9093: 637045
637832
637835
641153
652982
686541
[2026-02-05 10:26:36 UTC] USER=www-data EUID=0 PID=756328 ACTION=passthru ARGS=bash -c kill -9 637045
[2026-02-05 10:26:36 UTC] USER=www-data EUID=0 PID=756338 ACTION=passthru ARGS=bash -c kill -9 637832
[2026-02-05 10:26:36 UTC] USER=www-data EUID=0 PID=756349 ACTION=passthru ARGS=bash -c kill -9 637835
[2026-02-05 10:26:36 UTC] USER=www-data EUID=0 PID=756360 ACTION=passthru ARGS=bash -c kill -9 641153
[2026-02-05 10:26:36 UTC] USER=www-data EUID=0 PID=756369 ACTION=passthru ARGS=bash -c kill -9 652982
[2026-02-05 10:26:36 UTC] USER=www-data EUID=0 PID=756380 ACTION=passthru ARGS=bash -c kill -9 686541
  ✅ Legacy services stopped and rogue processes cleaned
🔓 Removing stale lock files...
[2026-02-05 10:26:41 UTC] USER=www-data EUID=0 PID=756570 ACTION=fsop ARGS=test -f /data/kafka/zone-universe-main-dev_coordinator-meta/.lock
[2026-02-05 10:26:41 UTC] USER=www-data EUID=0 PID=756580 ACTION=fsop ARGS=test -f /data/kafka/zone-universe-main-dev_coordinator-data/.lock
  ✅ Lock file check complete
🚀 Starting confluent-kraft-zone-universe-main-dev_coordinator.service ...
[2026-02-05 10:26:41 UTC] USER=www-data EUID=0 PID=756591 ACTION=passthru ARGS=systemctl enable confluent-kraft-zone-universe-main-dev_coordinator.service
[2026-02-05 10:26:42 UTC] USER=www-data EUID=0 PID=756648 ACTION=passthru ARGS=systemctl restart confluent-kraft-zone-universe-main-dev_coordinator.service
🔧 Patching shared Connect unit to follow KRaft broker...
[2026-02-05 10:26:45 UTC] USER=www-data EUID=0 PID=757230 ACTION=fsop ARGS=sed -i -e s|${FULL_ENV}|zone-universe-main-dev|g -e s|${IDENTIFIER}|coordinator|g -e s|${CONFIG_DIR}|/opt/kafka/config/zone-universe-main-dev/coordinator|g /etc/systemd/system/confluent-connect-zone-universe-main-dev_coordinator.service
[2026-02-05 10:26:45 UTC] USER=www-data EUID=0 PID=757272 ACTION=fsop ARGS=sed -i s|^Wants=.*|Wants=confluent-kraft-zone-universe-main-dev_coordinator.service| /etc/systemd/system/confluent-connect-zone-universe-main-dev_coordinator.service
[2026-02-05 10:26:45 UTC] USER=www-data EUID=0 PID=757284 ACTION=fsop ARGS=sed -i s|^ExecStart=.*|ExecStart=/opt/kafka/bin/connect-distributed.sh /opt/kafka/config/zone-universe-main-dev/coordinator/connect-distributed.properties| /etc/systemd/system/confluent-connect-zone-universe-main-dev_coordinator.service
[2026-02-05 10:26:45 UTC] USER=www-data EUID=0 PID=757300 ACTION=passthru ARGS=systemctl daemon-reload
  ✅ Connect unit patched
[2026-02-05 10:26:46 UTC] USER=www-data EUID=0 PID=757362 ACTION=fsop ARGS=test -f /opt/kafka/config/zone-universe-main-dev/coordinator/connect-distributed.properties
[2026-02-05 10:26:46 UTC] USER=www-data EUID=0 PID=757376 ACTION=fsop ARGS=ln -sf /opt/kafka/config/zone-universe-main-dev/coordinator/connect-distributed.properties /opt/kafka/config/connect-distributed.properties
⏳ Waiting for broker coordinator on SSL://eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com:9092 ...
⏳ Waiting for KRaft broker... (attempt 1, 0s/600s)
   Debug: Last error was: [2026-02-05 10:26:46 UTC] USER=www-data EUID=0 PID=757397 ACTION=passthru ARGS=bash -c timeout 5 sudo -u kafka /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server 'eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com:9092' --command-config '/etc/kafka/client-zone-universe-main-dev-coordinator.properties' describe --status
⏳ Still waiting... (attempt 10, 59s/600s)
⏳ Still waiting... (attempt 20, 124s/600s)
✅ coordinator responded after 169s (attempt 27)
---- server.properties (key lines) ----
[2026-02-05 10:30:02 UTC] USER=www-data EUID=0 PID=780604 ACTION=passthru ARGS=bash -c grep -E '^(listeners|advertised\.listeners|process\.roles|controller\.quorum\.voters|controller\.listener\.names|inter\.broker\.listener\.name|log\.dirs|metadata\.log\.dir)=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
listeners=SSL://10.100.1.52:9092,CONTROLLER://10.100.1.52:9093
advertised.listeners=SSL://eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com:9092
inter.broker.listener.name=SSL
log.dirs=/data/kafka/zone-universe-main-dev_coordinator-data
process.roles=broker,controller
controller.listener.names=CONTROLLER
controller.quorum.voters=1@10.100.1.52:9093
metadata.log.dir=/data/kafka/zone-universe-main-dev_coordinator-meta
---------------------------------------

✅ KRaft setup complete for zone-universe-main-dev_coordinator
  server.properties : /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
  data dir          : /data/kafka/zone-universe-main-dev_coordinator-data
  meta dir          : /data/kafka/zone-universe-main-dev_coordinator-meta
  systemd unit      : confluent-kraft-zone-universe-main-dev_coordinator.service

🔧 Kafka Configuration Modified:
  ✓ process.roles, node.id, controller.quorum.voters, controller.listener.names
  ✓ listeners (SSL + CONTROLLER) and advertised.listeners (FQDN fallback to IP)
  ✓ listener.security.protocol.map, inter.broker.listener.name
  ✓ log.dirs -> /data/kafka/zone-universe-main-dev_coordinator-data, metadata.log.dir -> /data/kafka/zone-universe-main-dev_coordinator-meta
  ✓ removed zookeeper.connect (if present)
  ✓ created/refreshed dedicated KRaft systemd unit
  ✓ patched shared Connect unit to follow KRaft broker
  ✓ symlinked /opt/kafka/config/zone-universe-main-dev/coordinator/connect-distributed.properties -> /opt/kafka/config/connect-distributed.properties (compat)

🔎 Check quorum:
  /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com:9092 --command-config /etc/kafka/client-zone-universe-main-dev-coordinator.properties describe --status

📋 Next steps:
  1) Review KRaft config:   sudo grep -E 'process.roles|node.id|controller|listeners|advertised.listeners|log.dirs|metadata.log.dir' /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
  2) Verify topics:         /opt/kafka/bin/kafka-topics.sh --bootstrap-server eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com:9092 --command-config /etc/kafka/client-zone-universe-main-dev-coordinator.properties --list

✅ KRaft metadata layer setup completed successfully

Next steps:
  1. Verify KRaft quorum status
  2. Create Kafka topics
  3. Configure Kafka Connect
[2026-02-05 10:30:03 UTC] USER=www-data EUID=0 PID=780767 ACTION=fsop ARGS=mkdir -p /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev
[INFO] Saved metadata mode to: /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/kafka_metadata_mode

════════════════════════════════════════════════════════════════════

✅ Kafka Metadata Layer Setup Complete

  Mode           : kraft
  Environment    : zone-universe-main-dev
  State saved    : /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/kafka_metadata_mode

  KRaft cluster.id: 8Ts5q7jcRsa8jOqSp19b1g

  Verify quorum:
    kafka-metadata-quorum.sh --bootstrap-server ... describe

════════════════════════════════════════════════════════════════════

[OK] ✅ Step 3 completed: 02-metadata.sh

[INFO] 📦 Step 4/13: restart kafka related services...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[2026-02-05 10:30:05 UTC] USER=www-data EUID=0 PID=781464 ACTION=fsop ARGS=test -f /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 10:30:05 UTC] USER=www-data EUID=0 PID=781507 ACTION=passthru ARGS=bash -c grep -E '^[[:space:]]*process\.roles=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties' | grep -Eq '(broker|controller)'
[INFO] 📋 Detected mode from server.properties: kraft
[2026-02-05 10:30:07 UTC] USER=www-data EUID=0 PID=781981 ACTION=passthru ARGS=systemctl stop confluent-connect-zone-universe-main-dev_coordinator.service
[2026-02-05 10:30:08 UTC] USER=www-data EUID=0 PID=782122 ACTION=passthru ARGS=systemctl stop confluent-kafka-zk-zone-universe-main-dev_coordinator.service
[2026-02-05 10:30:09 UTC] USER=www-data EUID=0 PID=782246 ACTION=passthru ARGS=systemctl stop confluent-zookeeper-zone-universe-main-dev_coordinator.service
Failed to stop confluent-zookeeper-zone-universe-main-dev_coordinator.service: Unit confluent-zookeeper-zone-universe-main-dev_coordinator.service not loaded.
[INFO] 🧹 Removing stale Kafka lock files...
[2026-02-05 10:30:13 UTC] USER=www-data EUID=0 PID=782778 ACTION=fsop ARGS=rm -f /var/lib/kafka/zone-universe-main-dev_coordinator-meta/.lock
[INFO] 🧹 Cleaning up orphaned processes on Kafka ports...
[2026-02-05 10:30:13 UTC] USER=www-data EUID=0 PID=782918 ACTION=passthru ARGS=bash -c 
for port in 9092 9093 8083 2181; do
  pids=$(lsof -ti tcp:$port 2>/dev/null || true)
  if [[ -n "$pids" ]]; then
    echo "   Killing orphaned processes on port $port: $pids"
    kill -9 $pids 2>/dev/null || true
    sleep 1
  fi
done

   Killing orphaned processes on port 9092: 756663
757867
757868
757869
757877
757985
758056
766162
767337
   Killing orphaned processes on port 9093: 757357
757359

🚀 Restarting Kafka components…
[INFO] 🚀 starting confluent-kraft-zone-universe-main-dev_coordinator.service…
[2026-02-05 10:30:20 UTC] USER=www-data EUID=0 PID=786231 ACTION=passthru ARGS=systemctl restart confluent-kraft-zone-universe-main-dev_coordinator.service
[INFO] 🚀 starting confluent-connect-zone-universe-main-dev_coordinator.service…
[2026-02-05 10:30:22 UTC] USER=www-data EUID=0 PID=787068 ACTION=passthru ARGS=systemctl restart confluent-connect-zone-universe-main-dev_coordinator.service

[INFO] ⏳ Waiting for Kafka broker readiness (FQDN: eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com, IP: 10.100.1.52) ...
[OK] ✅ Broker ready (attempt 1)
[OK] ✅ Port 9092 listening (Kafka Broker)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (1/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (2/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (3/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (4/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (5/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (6/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (7/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (8/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (9/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (10/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (11/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (12/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (13/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (14/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (15/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (16/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (17/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (18/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (19/40)
[OK] ✅ Port 8083 listening (Kafka Connect REST)
[INFO] ⏳ Waiting for Connect REST at https://eventbus-zone-universe-main-dev-kafka-connect.fastorder.com:8083 …
[OK] ✅ Connect REST is up (attempt 1)

📋 Reconciling Connect internal topics…
  [ok] connect-configs exists
  [ok] connect-offsets exists
  [ok] connect-status exists

═══════════════════════════════════════════════════════════════════
                           KAFKA SUMMARY
═══════════════════════════════════════════════════════════════════
Env: zone-universe-main-dev   Identifier: coordinator   Mode: kraft
Broker Unit : confluent-kraft-zone-universe-main-dev_coordinator.service  (status: active)
Connect Unit: confluent-connect-zone-universe-main-dev_coordinator.service (status: active)
Bootstrap   : eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com:9092
Connect URL : https://eventbus-zone-universe-main-dev-kafka-connect.fastorder.com:8083
═══════════════════════════════════════════════════════════════════
[OK] ✅ All required services are up.
[OK] ✅ Step 4 completed: 03-restart-kafka-related-services.sh

[INFO] 📦 Step 5/13: checking services...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[2026-02-05 10:33:02 UTC] USER=www-data EUID=0 PID=798571 ACTION=fsop ARGS=test -f /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 10:33:02 UTC] USER=www-data EUID=0 PID=798617 ACTION=passthru ARGS=bash -c grep -E '^[[:space:]]*process\.roles=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties' | grep -Eq '(broker|controller)'
[INFO] Detected mode from server.properties: kraft

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 1: Service status
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] confluent-kraft-zone-universe-main-dev_coordinator.service status: active
[WARN] confluent-kafka-zk-zone-universe-main-dev_coordinator.service present but should be stopped in KRaft
[WARN] confluent-zookeeper-zone-universe-main-dev_coordinator.service present but not required in KRaft
[OK] confluent-connect-zone-universe-main-dev_coordinator.service status: active

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 2: Port checks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Port 9092 listening (Kafka Broker)
[OK] ✅ Port 8083 listening (Kafka Connect REST)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 3: Broker readiness
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] Broker API responding (attempt 1)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 4: Kafka Connect REST
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] Connect REST responding (attempt 1)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Kafka Services Summary
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment : zone-universe-main-dev
Identifier  : coordinator
Mode        : kraft
Broker Unit : confluent-kraft-zone-universe-main-dev_coordinator.service  (status: active)
Connect Unit: confluent-connect-zone-universe-main-dev_coordinator.service (status: active)
Broker FQDN : eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com:9092
Broker IP   : eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com:9092
Connect URL : https://eventbus-zone-universe-main-dev-kafka-connect.fastorder.com:8083
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ All required services are reachable.
[INFO] Creating ACLs for Kafka Connect consumer groups...
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
	at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
	at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
	at kafka.admin.AclCommand$.main(AclCommand.scala:73)
	at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
	... 5 more

[INFO] Creating ACLs for Connect internal topics...
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
	at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
	at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
	at kafka.admin.AclCommand$.main(AclCommand.scala:73)
	at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
	... 5 more

Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
	at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
	at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
	at kafka.admin.AclCommand$.main(AclCommand.scala:73)
	at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
	... 5 more

Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
	at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
	at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
	at kafka.admin.AclCommand$.main(AclCommand.scala:73)
	at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
	... 5 more

Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
	at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
	at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
	at kafka.admin.AclCommand$.main(AclCommand.scala:73)
	at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
	... 5 more

Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
	at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
	at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
	at kafka.admin.AclCommand$.main(AclCommand.scala:73)
	at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
	... 5 more

[OK] ✅ Kafka Connect ACLs configured (deny-by-default mode)
[OK] ✅ Step 5 completed: 04-checking-services.sh

[INFO] 📦 Step 6/13: create audit topic...
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Creating Kafka Audit Topics
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Environment: zone-universe-main-dev
[INFO] Replication Factor: 1

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 AUDIT READINESS GATE - Preflight Checks
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] [Gate 1/5] Verifying DNS resolution...
[OK] ✅ Broker DNS: eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com -> 10.100.1.52
[OK] ✅ Connect DNS: eventbus-zone-universe-main-dev-kafka-connect.fastorder.com -> 10.100.1.110
[INFO] [Gate 2/5] Verifying TLS handshake...
[OK] ✅ TLS handshake: Broker certificate verified
[INFO] [Gate 3/5] Verifying Kafka Connect REST API...
[OK] ✅ Kafka Connect REST: Cluster ID = [2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635559 ACTION=passthru ARGS=bash -c cat /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/kafka_kraft_cluster_id
[INFO] [Gate 4/5] Verifying required internal topics...
[OK] ✅ Topic exists: connect-configs
[OK] ✅ Topic exists: connect-offsets
[OK] ✅ Topic exists: connect-status
[INFO] [Gate 5/5] Verifying broker metadata access...
[OK] ✅ Broker metadata: API versions accessible

[OK] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ AUDIT READINESS GATE: ALL CHECKS PASSED
[OK] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Waiting for Kafka to be ready...
[OK] ✅ Kafka is ready
[INFO] Creating audit topic: audit.events.zone.universe.main.dev
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic audit.events.zone.universe.main.dev.
[OK] ✅ Audit topic created: audit.events.zone.universe.main.dev
[OK] ✅ Topic verified: audit.events.zone.universe.main.dev
Topic: audit.events.zone.universe.main.dev	TopicId: fSLwJ6AoRSuK4PHDICM-fw	PartitionCount: 3	ReplicationFactor: 1	Configs: compression.type=lz4,min.insync.replicas=1,cleanup.policy=delete,segment.bytes=1073741824,retention.ms=7776000000,message.timestamp.type=LogAppendTime,segment.ms=604800000
	Topic: audit.events.zone.universe.main.dev	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: audit.events.zone.universe.main.dev	Partition: 1	Leader: 1	Replicas: 1	Isr: 1
	Topic: audit.events.zone.universe.main.dev	Partition: 2	Leader: 1	Replicas: 1	Isr: 1
[INFO] Creating audit producer credentials...
Completed updating config for user audit-producer-zone-universe-main-dev.
[OK] ✅ Audit producer user created: audit-producer-zone-universe-main-dev
[INFO] Creating ACLs for audit producer...
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.zone.universe.main.dev, patternType=LITERAL)`: 
 	(principal=User:audit-producer-zone-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.zone.universe.main.dev, patternType=LITERAL)`: 
 	(principal=User:audit-producer-zone-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.zone.universe.main.dev, patternType=LITERAL)`: 
 	(principal=User:audit-producer-zone-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.zone.universe.main.dev, patternType=LITERAL)`: 
 	(principal=User:audit-producer-zone-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:audit-producer-zone-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.zone.universe.main.dev, patternType=LITERAL)`: 
 	(principal=User:CN=kafka-client-zone-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:CN=kafka-client-zone-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.zone.universe.main.dev, patternType=LITERAL)`: 
 	(principal=User:CN=kafka-client-zone-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:CN=kafka-client-zone-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:audit-producer-zone-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:audit-producer-zone-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.zone.universe.main.dev, patternType=LITERAL)`: 
 	(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-zone-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-zone-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.zone.universe.main.dev, patternType=LITERAL)`: 
 	(principal=User:audit-producer-zone-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-zone-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-zone-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:CN=kafka-client-zone-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:CN=kafka-client-zone-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:audit-producer-zone-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.zone.universe.main.dev, patternType=LITERAL)`: 
 	(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-zone-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
	(principal=User:CN=kafka-client-zone-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.zone.universe.main.dev, patternType=LITERAL)`: 
 	(principal=User:audit-producer-zone-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-zone-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-zone-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-zone-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
	(principal=User:CN=kafka-client-zone-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:CN=kafka-client-zone-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:CN=kafka-client-zone-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
	(principal=User:audit-producer-zone-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.zone.universe.main.dev, patternType=LITERAL)`: 
 	(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
	(principal=User:*, host=*, operation=ALTER, permissionType=DENY) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.zone.universe.main.dev, patternType=LITERAL)`: 
 	(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
	(principal=User:audit-producer-zone-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-zone-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-zone-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-zone-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
	(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
	(principal=User:CN=kafka-client-zone-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:CN=kafka-client-zone-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:CN=kafka-client-zone-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
	(principal=User:audit-producer-zone-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW) 

[OK] ✅ ACLs configured (producer: write-only, sinks: read-only, immutability: protected)
[INFO] Storing audit producer credentials in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/eventbus/zone/universe/main/dev/kafka/audit/producer-leKdEU",
    "Name": "fastorder/eventbus/zone/universe/main/dev/kafka/audit/producer",
    "VersionId": "7fe04329-b7ca-4ec5-a4a6-1581ad4fb2b4"
}
[OK] ✅ Credentials stored in: fastorder/eventbus/zone/universe/main/dev/kafka/audit/producer
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuring S3 Sink for Audit Cold Storage
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                           PRE audit/
[INFO] S3 staging bucket exists: fastorder-audit-staging
[INFO] Data flow: Kafka → fastorder-audit-staging → (replication) → fastorder-audit-immutable
[INFO] Creating new S3 sink connector...
{"name":"audit-s3-sink-zone_universe_main_dev","config":{"connector.class":"io.confluent.connect.s3.S3SinkConnector","tasks.max":"1","topics":"audit.events.zone.universe.main.dev","topics.dir":"audit/zone-universe-main-dev","s3.bucket.name":"fastorder-audit-staging","s3.region":"me-central-1","s3.part.size":"5242880","flush.size":"1000","rotate.interval.ms":"3600000","rotate.schedule.interval.ms":"86400000","storage.class":"io.confluent.connect.s3.storage.S3Storage","format.class":"io.confluent.connect.s3.format.json.JsonFormat","partitioner.class":"io.confluent.connect.storage.partitioner.TimeBasedPartitioner","path.format":"'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH","partition.duration.ms":"3600000","locale":"en-US","timezone":"UTC","timestamp.extractor":"Record","key.converter":"org.apache.kafka.connect.json.JsonConverter","value.converter":"org.apache.kafka.connect.json.JsonConverter","key.converter.schemas.enable":"false","value.converter.schemas.enable":"false","behavior.on.null.values":"ignore","errors.tolerance":"all","errors.log.enable":"true","errors.log.include.messages":"true","name":"audit-s3-sink-zone_universe_main_dev"},"tasks":[],"type":"sink"}[OK] ✅ S3 Sink connector configured for audit cold storage
[INFO]   Staging Bucket: fastorder-audit-staging (Kafka Connect writes here)
[INFO]   Immutable Bucket: fastorder-audit-immutable (via S3 Replication)
[INFO]   Path: audit/zone-universe-main-dev/
[INFO]   Final Retention: WORM-enabled (Object Lock COMPLIANCE mode, 1-year)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Kafka Audit Topic Created Successfully
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Topic: audit.events.zone.universe.main.dev
[INFO] Partitions: 3
[INFO] Replication Factor: 1
[INFO] Retention: 90 days
[INFO] Producer: audit-producer-zone-universe-main-dev (write-only)

[INFO] Application Integration:
[INFO]   - Use credentials from: fastorder/eventbus/zone/universe/main/dev/kafka/audit/producer
[INFO]   - Connect to: eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com:9092
[INFO]   - Produce to: audit.events.zone.universe.main.dev
[INFO]   - Security: SASL_SSL (SCRAM-SHA-512)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📋 PCI-DSS Compliance Status
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 
[INFO] ✅ Kafka Hot Storage: 90 days (meets PCI-DSS 3-month immediate access)
[INFO] ✅ ACL Authorization: deny-by-default (allow.everyone.if.no.acl.found=false)
[INFO] ✅ Immutability: DENY DELETE/ALTER on audit topic
[INFO] ✅ S3 Cold Storage: fastorder-audit-immutable (Object Lock COMPLIANCE, 1-year)
[INFO] 
[INFO] S3 Audit Storage:
[INFO]   Bucket: s3://fastorder-audit-staging
[INFO]   Path: audit/zone-universe-main-dev/
[INFO]   Object Lock: COMPLIANCE mode, 1-year retention
[INFO]   Immutability: Objects cannot be deleted or modified for 1 year
[INFO] 
[INFO] Verify compliance with:
[INFO]   bash 04-eventbus/engine/kafka/steps/11-audit-compliance-check.sh
[INFO] 
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📋 AWS Roles Anywhere - Credential Refresh Setup
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 
[INFO] For S3 sink to write to S3, Kafka Connect needs AWS credentials.
[INFO] Use IAM Roles Anywhere with systemd timer for automatic refresh.
[INFO] 
[INFO] Required files:
[INFO]   Certificate: /etc/fastorder/rolesanywhere/client-bundle.crt
[INFO]   Private Key: /etc/fastorder/rolesanywhere/client.key
[INFO]   Helper: /usr/local/bin/aws_signing_helper
[INFO] 
[INFO] Systemd timer: kafka-aws-credential-refresh.timer
[INFO]   Runs every 30 minutes to refresh credentials to /var/lib/kafka/.aws/credentials
[INFO] 
[INFO] Verify timer is active:
[INFO]   systemctl status kafka-aws-credential-refresh.timer
[INFO] 
[INFO] Documentation: https://docs.aws.amazon.com/rolesanywhere/latest/userguide/

[OK] ✅ Step 6 completed: 05-create-audit-topic.sh

[INFO] 📦 Step 7/13: setup backups...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Kafka Backup Configuration
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Environment: zone-universe-main-dev

🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[INFO] 1️⃣ Creating S3 bucket for Kafka backups...
make_bucket failed: s3://fastorder-kafka-backups-zone-universe-main-dev An error occurred (AccessDenied) when calling the CreateBucket operation: User: arn:aws:iam::464621692046:user/fo-dev is not authorized to perform: s3:CreateBucket on resource: "arn:aws:s3:::fastorder-kafka-backups-zone-universe-main-dev" because no identity-based policy allows the s3:CreateBucket action

An error occurred (NoSuchBucket) when calling the PutBucketVersioning operation: The specified bucket does not exist

Parameter validation failed:
Unknown parameter in LifecycleConfiguration.Rules[0]: "Id", must be one of: Expiration, ID, Prefix, Filter, Status, Transitions, NoncurrentVersionTransitions, NoncurrentVersionExpiration, AbortIncompleteMultipartUpload
[OK] ✅ S3 bucket created: fastorder-kafka-backups-zone-universe-main-dev

[INFO] 2️⃣ Creating local backup directory...
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810047 ACTION=fsop ARGS=mkdir -p /var/backups/kafka/zone-universe-main-dev
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810056 ACTION=fsop ARGS=mkdir -p /var/backups/kafka/zone-universe-main-dev/topics
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810065 ACTION=fsop ARGS=mkdir -p /var/backups/kafka/zone-universe-main-dev/metadata
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810074 ACTION=fsop ARGS=mkdir -p /var/log/kafka/backups
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810083 ACTION=fsop ARGS=chown -R kafka:kafka /var/backups/kafka/zone-universe-main-dev
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810092 ACTION=fsop ARGS=chown -R kafka:kafka /var/log/kafka/backups
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810101 ACTION=fsop ARGS=chmod 750 /var/backups/kafka/zone-universe-main-dev
[OK] ✅ Local backup directory created

[INFO] 3️⃣ Creating topic backup script...
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810119 ACTION=fsop ARGS=sed -i s|__ENV_ID__|zone-universe-main-dev|g /usr/local/bin/kafka-backup-zone-universe-main-dev.sh
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810128 ACTION=fsop ARGS=sed -i s|__KAFKA_BROKER__|eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com:9092|g /usr/local/bin/kafka-backup-zone-universe-main-dev.sh
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810137 ACTION=fsop ARGS=sed -i s|__BACKUP_DIR__|/var/backups/kafka/zone-universe-main-dev|g /usr/local/bin/kafka-backup-zone-universe-main-dev.sh
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810146 ACTION=fsop ARGS=sed -i s|__S3_BUCKET__|fastorder-kafka-backups-zone-universe-main-dev|g /usr/local/bin/kafka-backup-zone-universe-main-dev.sh
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810155 ACTION=fsop ARGS=sed -i s|__S3_REGION__|me-central-1|g /usr/local/bin/kafka-backup-zone-universe-main-dev.sh
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810164 ACTION=fsop ARGS=chmod 750 /usr/local/bin/kafka-backup-zone-universe-main-dev.sh
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810173 ACTION=fsop ARGS=chown root:kafka /usr/local/bin/kafka-backup-zone-universe-main-dev.sh
[OK] ✅ Backup script created: /usr/local/bin/kafka-backup-zone-universe-main-dev.sh

[INFO] 4️⃣ Setting up cron jobs for automated backups...
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810191 ACTION=fsop ARGS=chmod 644 /etc/cron.d/kafka-backups-zone-universe-main-dev
[OK] ✅ Cron job configured: Daily backups at 2:00 AM

[INFO] 5️⃣ Creating restore documentation...
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810209 ACTION=fsop ARGS=sed -i s|__S3_BUCKET__|fastorder-kafka-backups-zone-universe-main-dev|g /var/backups/kafka/zone-universe-main-dev/RESTORE_INSTRUCTIONS.md
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810218 ACTION=fsop ARGS=sed -i s|__S3_REGION__|me-central-1|g /var/backups/kafka/zone-universe-main-dev/RESTORE_INSTRUCTIONS.md
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810227 ACTION=fsop ARGS=sed -i s|__KAFKA_BROKER__|eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com|g /var/backups/kafka/zone-universe-main-dev/RESTORE_INSTRUCTIONS.md
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810236 ACTION=fsop ARGS=chmod 644 /var/backups/kafka/zone-universe-main-dev/RESTORE_INSTRUCTIONS.md
[2026-02-05 10:35:13 UTC] USER=www-data EUID=0 PID=810245 ACTION=fsop ARGS=chown kafka:kafka /var/backups/kafka/zone-universe-main-dev/RESTORE_INSTRUCTIONS.md
[OK] ✅ Restore documentation created: /var/backups/kafka/zone-universe-main-dev/RESTORE_INSTRUCTIONS.md

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Kafka Backup Configured
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] S3 Bucket: fastorder-kafka-backups-zone-universe-main-dev
[INFO] Region: me-central-1
[INFO] Local backup dir: /var/backups/kafka/zone-universe-main-dev
[INFO] Schedule: Daily at 2:00 AM
[INFO] Script: /usr/local/bin/kafka-backup-zone-universe-main-dev.sh
[INFO] Restore docs: /var/backups/kafka/zone-universe-main-dev/RESTORE_INSTRUCTIONS.md

[WARN] ⚠️  Note: This backs up Kafka metadata only (topics, configs, offsets)
[WARN]    For full message data backup, configure Kafka Connect S3 Sink

[OK] ✅ Step 7 completed: 06-setup-backups.sh

[INFO] 📦 Step 8/13: monitoring setup...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Kafka Monitoring Integration for zone-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-zone-universe-main-dev
[OK]   Observability cell endpoints registered for zone-universe-main-dev
[OK]   ✓ Observability cell is ready

[INFO] 2️⃣ Setting up Kafka JMX exporter integration...
[INFO] JMX Exporter port calculated for zone-universe-main-dev: 9403 (offset: 95)
[INFO] Checking observability cell readiness: obs-zone-universe-main-dev
[OK]   Observability cell endpoints registered for zone-universe-main-dev
[INFO] Setting up Kafka JMX exporter for zone-universe-main-dev
[INFO] JMX Prometheus Java Agent already exists at /opt/kafka/libs/jmx_prometheus_javaagent.jar
[2026-02-05 10:35:14 UTC] USER=www-data EUID=0 PID=810290 ACTION=passthru ARGS=mv /tmp/jmx_exporter.yml /opt/kafka/config/jmx_exporter.yml
[2026-02-05 10:35:14 UTC] USER=www-data EUID=0 PID=810301 ACTION=passthru ARGS=chmod 644 /opt/kafka/config/jmx_exporter.yml
[OK]   JMX exporter configuration created at /opt/kafka/config/jmx_exporter.yml
[OK]   JMX exporter configuration created
[INFO] Configuring Kafka systemd services to use JMX exporter...
[2026-02-05 10:35:14 UTC] USER=www-data EUID=0 PID=810329 ACTION=fsop ARGS=test -f /etc/systemd/system/[2026-02-05
[INFO] All Kafka services already configured with JMX exporter
[OK]   Kafka JMX exporter integration complete
[INFO] Metrics endpoint: http://142.93.238.16:9403/metrics
[INFO] Prometheus will automatically scrape: https://metrics-zone-universe-main-dev.fastorder.com:9090
[INFO] View dashboards at: https://dashboards-zone-universe-main-dev.fastorder.com
[OK]   ✓ Kafka JMX exporter integration complete
[INFO] Configuring KAFKA_OPTS environment variable for kafka user...
[2026-02-05 10:35:14 UTC] USER=www-data EUID=0 PID=810352 ACTION=passthru ARGS=grep -q KAFKA_OPTS.*javaagent.*jmx_prometheus_javaagent.*=9403: /home/kafka/.bashrc
[2026-02-05 10:35:14 UTC] USER=www-data EUID=0 PID=810373 ACTION=passthru ARGS=sed -i /export KAFKA_OPTS=.*jmx_prometheus_javaagent/d /home/kafka/.bashrc
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[OK]   ✓ KAFKA_OPTS configured in /home/kafka/.bashrc (port 9403)
[INFO] 2.5️⃣ Enabling JMX exporter in Kafka systemd service...
[2026-02-05 10:35:14 UTC] USER=www-data EUID=0 PID=810407 ACTION=passthru ARGS=grep -q javaagent.*jmx_prometheus_javaagent /etc/systemd/system/confluent-kraft-zone-universe-main-dev_coordinator.service
[INFO] Updating confluent-kraft-zone-universe-main-dev_coordinator.service to enable JMX exporter...
[2026-02-05 10:35:14 UTC] USER=www-data EUID=0 PID=810428 ACTION=passthru ARGS=sed -i s|^Environment=KAFKA_OPTS=.*|Environment=KAFKA_OPTS=-javaagent:/opt/kafka/libs/jmx_prometheus_javaagent.jar=9403:/opt/kafka/config/jmx_exporter.yml| /etc/systemd/system/confluent-kraft-zone-universe-main-dev_coordinator.service
[OK]   ✓ Updated confluent-kraft-zone-universe-main-dev_coordinator.service
[INFO] Reloading systemd daemon and restarting Kafka services...
[2026-02-05 10:35:14 UTC] USER=www-data EUID=0 PID=810449 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 10:35:15 UTC] USER=www-data EUID=0 PID=810542 ACTION=passthru ARGS=systemctl is-active --quiet confluent-kraft-zone-universe-main-dev_coordinator
[INFO] Restarting confluent-kraft-zone-universe-main-dev_coordinator...
[2026-02-05 10:35:15 UTC] USER=www-data EUID=0 PID=810579 ACTION=passthru ARGS=systemctl restart confluent-kraft-zone-universe-main-dev_coordinator
[2026-02-05 10:35:20 UTC] USER=www-data EUID=0 PID=811204 ACTION=passthru ARGS=systemctl is-active --quiet confluent-kraft-zone-universe-main-dev_coordinator
[OK]   ✓ confluent-kraft-zone-universe-main-dev_coordinator restarted successfully
[OK]   ✓ JMX exporter enabled in Kafka systemd services
[INFO] 2.6️⃣ Configuring Prometheus to scrape Kafka metrics...
[2026-02-05 10:35:20 UTC] USER=www-data EUID=0 PID=811235 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml
[INFO] Adding Kafka scrape target to Prometheus configuration...
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[2026-02-05 10:35:20 UTC] USER=www-data EUID=0 PID=811273 ACTION=passthru ARGS=sed -i /# Prometheus self-monitoring/r /tmp/prometheus_kafka_add.yml /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml
[ERROR] Invalid Prometheus configuration - rolling back
[2026-02-05 10:35:20 UTC] USER=www-data EUID=0 PID=811315 ACTION=passthru ARGS=sed -i /job_name: 'kafka'/,+6d /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml
[2026-02-05 10:35:21 UTC] USER=www-data EUID=0 PID=811349 ACTION=fsop ARGS=rm -f /tmp/prometheus_kafka_add.yml

[INFO] 3️⃣ Registering Kafka nodes to monitoring database...
[INFO] Detected Kafka version: 3.9.1
[INFO] Registering Kafka Broker to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Kafka Broker
[INFO]   Identifier:        zone-universe-main-dev-broker-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                142.93.238.16
[INFO]   Port:              9092
[INFO]   FQDN:              eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 447d4dd5-e271-4d9e-91d6-bc2f569019ee
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   ✓ Kafka broker registered
[INFO] Registering Kafka Connect to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Kafka Connect
[INFO]   Identifier:        zone-universe-main-dev-connect-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8083
[INFO]   FQDN:              eventbus-zone-universe-main-dev-kafka-connect.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 23c140b3-b995-4158-a535-f8c0a04adfa6
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   ✓ Kafka Connect registered
[INFO] Schema Registry not running, skipping registration

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Kafka Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Metrics: http://localhost:9403/metrics
[INFO] Prometheus: https://metrics-zone-universe-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-zone-universe-main-dev.fastorder.com
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Step 8 completed: 10-monitoring-setup.sh

[INFO] 📦 Step 9/13: audit compliance check...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  KAFKA AUDIT COMPLIANCE DASHBOARD - PCI-DSS Verification
  Environment: zone-universe-main-dev
  Timestamp: 2026-02-05 10:35:28 UTC
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[1/5] Kafka Deny-by-Default ACL Posture
     Requirement: allow.everyone.if.no.acl.found=false
PASS Deny-by-default is ENABLED (allow.everyone.if.no.acl.found=false)

[2/5] Audit Topic Hot Retention (90 days)
     Requirement: retention.ms >= 7776000000 (90 days)
PASS Retention is 90 days (7776000000 ms)

[3/5] Kafka Connect S3 Sink Status
     Requirement: Connector and all tasks RUNNING
FAIL Connector RUNNING but tasks FAILED: FAILED

[4/5] S3 Freshness Evidence
     Requirement: Newest object < 120 minutes old
WARN No objects found in s3://fastorder-audit-immutable/audit/zone-universe-main-dev/
     This may be normal if no audit events have been generated yet

[5/5] S3 Object Lock Immutability
     Requirement: COMPLIANCE mode with 1-year retention
FAIL Cannot verify Object Lock configuration - access denied

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  COMPLIANCE SUMMARY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  COMPLIANCE ISSUES DETECTED
  Passed: 2/5
  Failed: 3/5

  Review failed checks above and remediate.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[OK] ✅ Step 9 completed: 11-audit-compliance-check.sh

[INFO] 📦 Step 10/13: audit canary test...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  KAFKA AUDIT CANARY TEST - End-to-End Verification
  Environment: zone-universe-main-dev
  Canary ID: canary-1770287743-813155
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[Step 1/4] Producing canary audit event to Kafka
     Topic: audit.events.zone.universe.main.dev
     Event: canary-1770287743-813155
     Failed to produce event
     (Topic may not exist yet - normal during initial setup)
[OK] ✅ Step 10 completed: 12-audit-canary-test.sh

[INFO] 📦 Step 11/13: setup audit s3 staging...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
════════════════════════════════════════════════════════════════════════
  Kafka Audit S3 Staging + Replication Setup
════════════════════════════════════════════════════════════════════════

  Staging Bucket:   fastorder-audit-staging
  Immutable Bucket: fastorder-audit-immutable
  Region:           me-central-1
  Environment:      --auto

════════════════════════════════════════════════════════════════════════

[INFO] 1️⃣  Checking AWS credentials...
[WARN] No AWS credentials found - skipping S3 staging setup

To configure S3 audit storage later, add credentials to /home/ab/.aws/credentials:
[admin]
aws_access_key_id = AKIA...
aws_secret_access_key = ...

Then run: AWS_PROFILE=admin /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps/13-setup-audit-s3-staging.sh --auto
[OK] ✅ Step 11 completed: 13-setup-audit-s3-staging.sh

[INFO] 📦 Step 12/13: install ksqldb...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ksqlDB Installation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Identifier:  --auto
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Allocating new VM_IP for ksqlDB: 10.100.1.251
🔧 Adding VM_IP 10.100.1.251 to loopback interface...
[2026-02-05 10:35:49 UTC] USER=www-data EUID=0 PID=813878 ACTION=configure-network-interface ARGS=lo 10.100.1.251
✓ lo <- 10.100.1.251
[2026-02-05 10:35:50 UTC] USER=www-data EUID=0 PID=813913 ACTION=fsop ARGS=tee -a /etc/hosts
10.100.1.251 eventbus-zone-universe-main-dev-ksqldb---auto.fastorder.com
  VM_IP:     10.100.1.251
  FQDN:      eventbus-zone-universe-main-dev-ksqldb---auto.fastorder.com

📦 Step 1: Checking Confluent Platform installation...
✅ ksqlDB already installed (version: )

📁 Step 2: Creating directories...
[2026-02-05 10:35:54 UTC] USER=www-data EUID=0 PID=814103 ACTION=fsop ARGS=mkdir -p /var/lib/ksqldb/zone-universe-main-dev/--auto
[2026-02-05 10:35:54 UTC] USER=www-data EUID=0 PID=814126 ACTION=fsop ARGS=mkdir -p /var/log/ksqldb/zone-universe-main-dev/--auto
[2026-02-05 10:35:55 UTC] USER=www-data EUID=0 PID=814147 ACTION=fsop ARGS=mkdir -p /etc/ksqldb/zone-universe-main-dev/--auto
[2026-02-05 10:35:55 UTC] USER=www-data EUID=0 PID=814168 ACTION=fsop ARGS=chown -R kafka:kafka /var/lib/ksqldb/zone-universe-main-dev/--auto /var/log/ksqldb/zone-universe-main-dev/--auto /etc/ksqldb/zone-universe-main-dev/--auto
✅ Directories created

⚙️  Step 3: Generating ksqlDB configuration...
[2026-02-05 10:35:55 UTC] USER=www-data EUID=0 PID=814191 ACTION=fsop ARGS=mv /tmp/ksql-server-zone-universe-main-dev.properties /etc/ksqldb/zone-universe-main-dev/--auto/ksql-server.properties
[2026-02-05 10:35:55 UTC] USER=www-data EUID=0 PID=814212 ACTION=fsop ARGS=chown kafka:kafka /etc/ksqldb/zone-universe-main-dev/--auto/ksql-server.properties
[2026-02-05 10:35:55 UTC] USER=www-data EUID=0 PID=814235 ACTION=fsop ARGS=chmod 640 /etc/ksqldb/zone-universe-main-dev/--auto/ksql-server.properties
✅ Configuration generated: /etc/ksqldb/zone-universe-main-dev/--auto/ksql-server.properties

🔧 Step 4: Creating systemd service...
[2026-02-05 10:35:55 UTC] USER=www-data EUID=0 PID=814258 ACTION=fsop ARGS=mv /tmp/ksqldb-zone-universe-main-dev---auto.service /etc/systemd/system/ksqldb-zone-universe-main-dev---auto.service
[2026-02-05 10:35:55 UTC] USER=www-data EUID=0 PID=814279 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 10:35:56 UTC] USER=www-data EUID=0 PID=814366 ACTION=passthru ARGS=systemctl enable ksqldb-zone-universe-main-dev---auto.service
Created symlink /etc/systemd/system/multi-user.target.wants/ksqldb-zone-universe-main-dev---auto.service → /etc/systemd/system/ksqldb-zone-universe-main-dev---auto.service.
✅ Systemd service created: ksqldb-zone-universe-main-dev---auto.service

🚀 Step 5: Starting ksqlDB service...
🔍 Checking Kafka broker connectivity...
✅ Kafka broker is accessible
[2026-02-05 10:35:57 UTC] USER=www-data EUID=0 PID=814445 ACTION=passthru ARGS=systemctl start ksqldb-zone-universe-main-dev---auto.service
✅ ksqlDB service started
⏳ Waiting for ksqlDB to be ready...
..............................

🔍 Step 6: Verifying installation...

📊 Service Status:
[2026-02-05 10:36:58 UTC] USER=www-data EUID=0 PID=816952 ACTION=passthru ARGS=systemctl status ksqldb-zone-universe-main-dev---auto.service --no-pager -l

📊 ksqlDB Info:
⚠️  ksqlDB not responding yet (may still be starting)

📡 Step 7: Registering ksqlDB to Observability API...
🔄 Registering ksqlDB node to observability dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       ksqlDB
[INFO]   Identifier:        zone-universe-main-dev-ksqldb---auto
[INFO]   Identifier Parent: eventbus
[INFO]   IP:                10.100.1.251
[INFO]   Port:              8088
[INFO]   FQDN:              eventbus-zone-universe-main-dev-ksqldb---auto.fastorder.com
[INFO]   Status:            starting
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 7e966311-6834-4a03-a8c8-f9c2ea668376
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
✅ ksqlDB registered successfully

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ksqlDB Installation Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Service:  ksqldb-zone-universe-main-dev---auto
  VM_IP:    10.100.1.251
  FQDN:     eventbus-zone-universe-main-dev-ksqldb---auto.fastorder.com
  Port:     8088
  Config:   /etc/ksqldb/zone-universe-main-dev/--auto/ksql-server.properties
  Data:     /var/lib/ksqldb/zone-universe-main-dev/--auto
  Logs:     /var/log/ksqldb/zone-universe-main-dev/--auto

  Dashboard:
    https://skeleton.dev.fastorder.com/dashboard/monitoring/environment2/<env-id>/service/ksqldb

  CLI Access (with SSL):
    ksql --ssl https://eventbus-zone-universe-main-dev-ksqldb---auto.fastorder.com:8088

  REST API (HTTPS):
    curl -k https://eventbus-zone-universe-main-dev-ksqldb---auto.fastorder.com:8088/info
    curl -k https://eventbus-zone-universe-main-dev-ksqldb---auto.fastorder.com:8088/ksql -H 'Content-Type: application/vnd.ksql.v1+json' -d '{"ksql": "SHOW STREAMS;"}'
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Step 12 completed: 20-install-ksqldb.sh

[INFO] 📦 Step 13/13: update www data certs...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
📋 Post-Kafka Setup: Updating www-data Kafka certificates...
   Environment: zone-universe-main-dev
   Source: /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/zone-universe-main-dev

✓ Kafka certificates found
✓ www-data user exists

[2026-02-05 10:36:59 UTC] USER=www-data EUID=0 PID=817063 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:36:59 UTC] USER=www-data EUID=0 PID=817081 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 10:36:59 UTC] USER=www-data EUID=0 PID=817090 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 10:36:59 UTC] USER=www-data EUID=0 PID=817099 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 10:36:59 UTC] USER=www-data EUID=0 PID=817108 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:36:59 UTC] USER=www-data EUID=0 PID=817120 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:36:59 UTC] USER=www-data EUID=0 PID=817130 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:36:59 UTC] USER=www-data EUID=0 PID=817139 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem

✅ Kafka certificate symlinks created for www-data
   PHP Kafka consumers can now use:
   - ssl.ca.location: /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
   - ssl.certificate.location: /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
   - ssl.key.location: /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem

✓ Post-Kafka setup complete
[OK] ✅ Step 13 completed: 99-update-www-data-certs.sh


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Kafka setup completed successfully!
[OK] Executed all 13 steps
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Environment: zone-universe-main-dev
[INFO] Service: zone
[INFO] Zone: universe
[INFO] Branch: main
[INFO] Env: dev
[INFO] Registering Kafka nodes via API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Kafka
[INFO]   Identifier:        zone-universe-main-dev_coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.52
[INFO]   Port:              9092
[INFO]   FQDN:              eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 02cfff26-4fe0-4bae-837d-34cc44821571
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Kafka Connect
[INFO]   Identifier:        zone-universe-main-dev_coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.110
[INFO]   Port:              8083
[INFO]   FQDN:              eventbus-zone-universe-main-dev-kafka-connect.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 2aa16713-7a1f-479a-bdb1-9a022e0bb97d
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK] ✔ Kafka node registration completed
[INFO] Setting up Kafka observability integration...
[INFO] Checking observability cell readiness: obs-zone-universe-main-dev
[OK] Observability cell endpoints registered for zone-universe-main-dev
[INFO] Observability cell verified for zone-universe-main-dev
[INFO] Monitoring will be configured after Kafka deployment (step 10-monitoring-setup.sh)
[INFO] Cleaning up temporary files...
[INFO] Starting cleanup of temporary files...
[INFO] Cleaning up SSL temp files for zone-universe-main-dev...
[INFO] Cleaning up old provisioning logs...
[INFO] Cleaning up old configuration backups...
[OK] ✔ Cleanup completed

✓ ✅ Event bus infrastructure (kafka) setup completed successfully

[INFO] Using database engine from DB_ENGINE environment variable: postgresql
[INFO] Cleaning up any existing locks...

Starting database engine: postgresql
═══════════════════════════════════════════════

[INFO] Using environment from web interface: zone-universe-main-dev
[2026-02-05 10:37:01] Using web-provided environment: zone-universe-main-dev
[2026-02-05 10:37:01] Service: zone, Zone: universe, Branch: main, Env: dev
✓ Environment initialized successfully (mode: general)
[INFO] Checking observability cell readiness: obs-zone-universe-main-dev
[OK]   Observability cell endpoints registered for zone-universe-main-dev
[INFO] Observability cell verified for zone-universe-main-dev
[INFO] Monitoring will be configured after PostgreSQL deployment (step 10-monitoring-setup.sh)
[INFO] Citus mode ENABLED
[INFO] → Coordinator + 1 worker(s) + 1 standby node(s) per worker
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up coordinator (Citus control plane)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-02-05 10:37:02 UTC] USER=unknown EUID=33 PID=817499 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 10:37:02 UTC] USER=unknown EUID=33 PID=817506 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 10:37:02 UTC] USER=unknown EUID=33 PID=817513 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-02-05 10:37:02 UTC] USER=unknown EUID=33 PID=817523 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 10:37:02 UTC] USER=unknown EUID=33 PID=817530 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-02-05 10:37:02 UTC] USER=unknown EUID=33 PID=817537 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for zone in universe-dev...
[INFO] Environment: zone-universe-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172 JOB_UUID=20cfe3cc-b016-4175-bac8-362f4003c72d

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: zone-universe-main-dev
[INFO] Identifier: coordinator
[INFO] PostgreSQL IP: 10.100.1.114
[INFO] Primary hostname: db-zone-universe-main-dev-postgresql-coordinator.fastorder.com

[INFO] Adding /etc/hosts entries for coordinator...
[INFO]   1. db-zone-universe-main-dev-postgresql.fastorder.com → 10.100.1.114 (primary/short)
[INFO]   2. db-zone-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.114 (compatibility)

[INFO]   ➕ Adding db-zone-universe-main-dev-postgresql.fastorder.com → 10.100.1.114
✅     ✅ Added: db-zone-universe-main-dev-postgresql.fastorder.com → 10.100.1.114
[INFO]   ➕ Adding db-zone-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.114
✅     ✅ Added: db-zone-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.114

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.114    db-zone-universe-main-dev-postgresql.fastorder.com
  10.100.1.114    db-zone-universe-main-dev-postgresql-coordinator.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-zone-universe-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      zone-universe-main-dev.fastorder.com
  Internal IP: 10.100.1.55 (from topology: db-coordinator-postgresql)
  Shared IP:   142.93.238.16 (legacy/public, also included in SANs)
  Coordinator variants:
    - db-zone-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator and /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:37:10 UTC] USER=www-data EUID=0 PID=818014 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-02-05 10:37:10 UTC] USER=www-data EUID=0 PID=818024 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-817792
[2026-02-05 10:37:10 UTC] USER=www-data EUID=0 PID=818033 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-817792/ra_root.crt
[2026-02-05 10:37:10 UTC] USER=www-data EUID=0 PID=818042 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-817792/ra_root.key
[2026-02-05 10:37:10 UTC] USER=www-data EUID=0 PID=818051 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-817792/ra_root.crt
[2026-02-05 10:37:10 UTC] USER=www-data EUID=0 PID=818072 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-817792/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-zone-universe-main-dev-postgresql-coordinator.fastorder.com
[2026-02-05 10:37:14 UTC] USER=www-data EUID=0 PID=818248 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-817792/server.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.key
[2026-02-05 10:37:14 UTC] USER=www-data EUID=0 PID=818261 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-817792/server.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.crt
[2026-02-05 10:37:14 UTC] USER=www-data EUID=0 PID=818270 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-02-05 10:37:14 UTC] USER=www-data EUID=0 PID=818282 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-817792/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:14 UTC] USER=www-data EUID=0 PID=818291 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:14 UTC] USER=www-data EUID=0 PID=818300 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 10:37:15 UTC] USER=www-data EUID=0 PID=818321 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.key
[2026-02-05 10:37:15 UTC] USER=www-data EUID=0 PID=818330 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.key
[2026-02-05 10:37:15 UTC] USER=www-data EUID=0 PID=818339 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.crt
[2026-02-05 10:37:15 UTC] USER=www-data EUID=0 PID=818348 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.crt
[2026-02-05 10:37:15 UTC] USER=www-data EUID=0 PID=818360 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:37:15 UTC] USER=www-data EUID=0 PID=818370 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-zone-universe-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-zone-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:zone-universe-main-dev.fastorder.com, DNS:db-zone-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:db-zone-universe-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-zone-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-zone-universe-main-dev-postgresql.fastorder.com, IP Address:10.100.1.55, IP Address:127.0.0.1, IP Address:142.93.238.16
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: zone-universe-main-dev
Node:        coordinator
Primary CN:  db-zone-universe-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@zone-universe-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-zone-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-zone-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:37:16 UTC] USER=www-data EUID=0 PID=818466 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-02-05 10:37:16 UTC] USER=www-data EUID=0 PID=818475 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 10:37:16 UTC] USER=www-data EUID=0 PID=818484 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-02-05 10:37:16 UTC] USER=www-data EUID=0 PID=818493 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 10:37:16 UTC] USER=www-data EUID=0 PID=818502 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:37:16 UTC] USER=www-data EUID=0 PID=818522 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:37:16 UTC] USER=www-data EUID=0 PID=818531 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:37:16 UTC] USER=www-data EUID=0 PID=818540 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key
[2026-02-05 10:37:16 UTC] USER=www-data EUID=0 PID=818549 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.crt
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818558 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818567 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818576 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818587 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818612 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818625 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818634 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818643 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818652 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818661 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818679 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818691 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818722 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818731 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818740 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818753 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818764 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818773 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818782 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818803 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818812 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818821 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818832 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_pk8.der /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 10:37:17 UTC] USER=www-data EUID=0 PID=818842 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=818852 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=818861 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=818880 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=818899 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=818908 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=818917 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=818926 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=818935 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=818944 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=818953 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=818962 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=818971 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_pk8.der /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=818981 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=818991 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819000 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819009 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819018 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819027 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819036 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819045 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819054 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819063 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819072 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819084 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819093 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_pk8.der /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819103 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819113 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819131 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819140 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819149 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819158 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key
[2026-02-05 10:37:18 UTC] USER=www-data EUID=0 PID=819167 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt
[2026-02-05 10:37:19 UTC] USER=www-data EUID=0 PID=819176 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:19 UTC] USER=www-data EUID=0 PID=819185 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:37:19 UTC] USER=www-data EUID=0 PID=819194 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 10:37:19 UTC] USER=www-data EUID=0 PID=819203 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 10:37:19 UTC] USER=www-data EUID=0 PID=819215 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_pk8.der /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 10:37:19 UTC] USER=www-data EUID=0 PID=819225 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:37:19 UTC] USER=www-data EUID=0 PID=819243 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:37:19 UTC] USER=www-data EUID=0 PID=819252 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:37:19 UTC] USER=www-data EUID=0 PID=819261 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 10:37:19 UTC] USER=www-data EUID=0 PID=819270 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 10:37:19 UTC] USER=www-data EUID=0 PID=819279 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 10:37:19 UTC] USER=www-data EUID=0 PID=819288 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:37:19 UTC] USER=www-data EUID=0 PID=819298 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:37:19 UTC] USER=www-data EUID=0 PID=819308 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:37:19 UTC] USER=www-data EUID=0 PID=819326 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: zone-universe-main-dev
User: postgres
Node: coordinator
FQDN: db-zone-universe-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-zone-universe-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-zone-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819377 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819387 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819397 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819406 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819416 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819442 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819455 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819469 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819480 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.crt
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819499 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819522 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819546 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819555 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819564 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819595 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819629 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819647 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:20 UTC] USER=www-data EUID=0 PID=819668 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:37:21 UTC] USER=www-data EUID=0 PID=819690 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key
[2026-02-05 10:37:21 UTC] USER=www-data EUID=0 PID=819767 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 10:37:21 UTC] USER=www-data EUID=0 PID=819842 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:21 UTC] USER=www-data EUID=0 PID=819869 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:21 UTC] USER=www-data EUID=0 PID=819933 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:21 UTC] USER=www-data EUID=0 PID=819955 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:37:21 UTC] USER=www-data EUID=0 PID=819973 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:37:21 UTC] USER=www-data EUID=0 PID=819997 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:21 UTC] USER=www-data EUID=0 PID=820051 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key
[2026-02-05 10:37:21 UTC] USER=www-data EUID=0 PID=820071 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt
[2026-02-05 10:37:21 UTC] USER=www-data EUID=0 PID=820133 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:22 UTC] USER=www-data EUID=0 PID=820175 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:37:22 UTC] USER=www-data EUID=0 PID=820214 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 10:37:22 UTC] USER=www-data EUID=0 PID=820247 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 10:37:22 UTC] USER=www-data EUID=0 PID=820272 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_pk8.der /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 10:37:22 UTC] USER=www-data EUID=0 PID=820300 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:22 UTC] USER=www-data EUID=0 PID=820370 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:37:22 UTC] USER=www-data EUID=0 PID=820393 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:37:22 UTC] USER=www-data EUID=0 PID=820416 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:22 UTC] USER=www-data EUID=0 PID=820441 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:22 UTC] USER=www-data EUID=0 PID=820470 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key
[2026-02-05 10:37:22 UTC] USER=www-data EUID=0 PID=820505 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt
[2026-02-05 10:37:22 UTC] USER=www-data EUID=0 PID=820523 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:22 UTC] USER=www-data EUID=0 PID=820547 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:37:22 UTC] USER=www-data EUID=0 PID=820571 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 10:37:22 UTC] USER=www-data EUID=0 PID=820587 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 10:37:22 UTC] USER=www-data EUID=0 PID=820613 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_pk8.der /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 10:37:23 UTC] USER=www-data EUID=0 PID=820636 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:23 UTC] USER=www-data EUID=0 PID=820657 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:23 UTC] USER=www-data EUID=0 PID=820675 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:37:23 UTC] USER=www-data EUID=0 PID=820694 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:37:23 UTC] USER=www-data EUID=0 PID=820721 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:23 UTC] USER=www-data EUID=0 PID=820766 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:23 UTC] USER=www-data EUID=0 PID=820793 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key
[2026-02-05 10:37:23 UTC] USER=www-data EUID=0 PID=820841 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:23 UTC] USER=www-data EUID=0 PID=820860 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:37:23 UTC] USER=www-data EUID=0 PID=820880 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 10:37:23 UTC] USER=www-data EUID=0 PID=820902 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 10:37:23 UTC] USER=www-data EUID=0 PID=820924 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_pk8.der /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 10:37:23 UTC] USER=www-data EUID=0 PID=820964 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:23 UTC] USER=www-data EUID=0 PID=820981 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:23 UTC] USER=www-data EUID=0 PID=821017 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:37:24 UTC] USER=www-data EUID=0 PID=821050 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:37:24 UTC] USER=www-data EUID=0 PID=821083 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:24 UTC] USER=www-data EUID=0 PID=821102 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:37:24 UTC] USER=www-data EUID=0 PID=821117 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key
[2026-02-05 10:37:24 UTC] USER=www-data EUID=0 PID=821137 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt
[2026-02-05 10:37:24 UTC] USER=www-data EUID=0 PID=821156 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:24 UTC] USER=www-data EUID=0 PID=821174 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:37:24 UTC] USER=www-data EUID=0 PID=821202 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 10:37:24 UTC] USER=www-data EUID=0 PID=821236 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 10:37:24 UTC] USER=www-data EUID=0 PID=821253 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres_pk8.der /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 10:37:24 UTC] USER=www-data EUID=0 PID=821270 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:37:24 UTC] USER=www-data EUID=0 PID=821289 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:37:24 UTC] USER=www-data EUID=0 PID=821309 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:37:24 UTC] USER=www-data EUID=0 PID=821326 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 10:37:24 UTC] USER=www-data EUID=0 PID=821349 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
   ✅ Symlinked client-key.pem
[2026-02-05 10:37:24 UTC] USER=www-data EUID=0 PID=821388 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:37:24 UTC] USER=www-data EUID=0 PID=821415 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:37:25 UTC] USER=www-data EUID=0 PID=821433 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:37:25 UTC] USER=www-data EUID=0 PID=821453 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: zone-universe-main-dev
User: postgres
Node: coordinator
FQDN: db-zone-universe-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-zone-universe-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-coordinator-postgresql environment: db-zone-universe-main-dev-postgresql-coordinator.fastorder.com (10.100.1.55)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.55
[INFO] Environment: zone-universe-main-dev
[INFO] Identifier: coordinator
[INFO] Data dir:   /data/postgresql/17/zone-universe-main-dev/coordinator
[INFO] Port:       5432
[INFO] Hostname:   db-zone-universe-main-dev-postgresql-coordinator
[2026-02-05 10:37:28 UTC] USER=www-data EUID=0 PID=822630 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:37:28 UTC] USER=www-data EUID=0 PID=822729 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-zone-universe-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      zone-universe-main-dev.fastorder.com
  Internal IP: 10.100.1.55 (from topology: db-coordinator-postgresql)
  Shared IP:   142.93.238.16 (legacy/public, also included in SANs)
  Coordinator variants:
    - db-zone-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 10:37:29 UTC] USER=www-data EUID=0 PID=823005 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator and /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-02-05 10:37:29 UTC] USER=www-data EUID=0 PID=823099 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-822744
[2026-02-05 10:37:30 UTC] USER=www-data EUID=0 PID=823143 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-822744/ra_root.crt
[2026-02-05 10:37:30 UTC] USER=www-data EUID=0 PID=823174 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-822744/ra_root.key
[2026-02-05 10:37:30 UTC] USER=www-data EUID=0 PID=823195 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-822744/ra_root.crt
[2026-02-05 10:37:30 UTC] USER=www-data EUID=0 PID=823217 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-822744/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-zone-universe-main-dev-postgresql-coordinator.fastorder.com
[2026-02-05 10:37:38 UTC] USER=www-data EUID=0 PID=825925 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-822744/server.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.key
[2026-02-05 10:37:38 UTC] USER=www-data EUID=0 PID=825954 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-822744/server.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-02-05 10:37:38 UTC] USER=www-data EUID=0 PID=826025 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-822744/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:38 UTC] USER=www-data EUID=0 PID=826062 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:38 UTC] USER=www-data EUID=0 PID=826092 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:37:38 UTC] USER=www-data EUID=0 PID=826125 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 10:37:38 UTC] USER=www-data EUID=0 PID=826152 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.key
[2026-02-05 10:37:38 UTC] USER=www-data EUID=0 PID=826178 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.key
[2026-02-05 10:37:38 UTC] USER=www-data EUID=0 PID=826213 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.crt
[2026-02-05 10:37:38 UTC] USER=www-data EUID=0 PID=826240 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.crt
[2026-02-05 10:37:39 UTC] USER=www-data EUID=0 PID=826276 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:37:39 UTC] USER=www-data EUID=0 PID=826303 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-zone-universe-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-zone-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:zone-universe-main-dev.fastorder.com, DNS:db-zone-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:db-zone-universe-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-zone-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-zone-universe-main-dev-postgresql.fastorder.com, IP Address:10.100.1.55, IP Address:127.0.0.1, IP Address:142.93.238.16
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: zone-universe-main-dev
Node:        coordinator
Primary CN:  db-zone-universe-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@zone-universe-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-zone-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-02-05 10:37:39 UTC] USER=www-data EUID=0 PID=826390 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.crt
[2026-02-05 10:37:39 UTC] USER=www-data EUID=0 PID=826413 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.key
[2026-02-05 10:37:39 UTC] USER=www-data EUID=0 PID=826440 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-02-05 10:37:39 UTC] USER=www-data EUID=0 PID=826464 ACTION=passthru ARGS=systemctl stop postgresql@zone-universe-main-dev-coordinator.service
[2026-02-05 10:37:39 UTC] USER=www-data EUID=0 PID=826493 ACTION=passthru ARGS=systemctl stop postgresql
[OK]   No conflicting Postgres left on port 5432
[OK]   Generated new postgres password for initdb
[2026-02-05 10:38:08 UTC] USER=www-data EUID=0 PID=827697 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.73Riv1
[2026-02-05 10:38:08 UTC] USER=www-data EUID=0 PID=827727 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.73Riv1
[2026-02-05 10:38:08 UTC] USER=www-data EUID=0 PID=827749 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/zone-universe-main-dev
[2026-02-05 10:38:08 UTC] USER=www-data EUID=0 PID=827800 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/zone-universe-main-dev
[2026-02-05 10:38:08 UTC] USER=www-data EUID=0 PID=827831 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/zone-universe-main-dev
[INFO] Initializing cluster in /data/postgresql/17/zone-universe-main-dev/coordinator (SCRAM; pwfile)
[2026-02-05 10:38:09 UTC] USER=www-data EUID=0 PID=827853 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/zone-universe-main-dev/coordinator
[2026-02-05 10:38:09 UTC] USER=www-data EUID=0 PID=827876 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/zone-universe-main-dev/coordinator
[2026-02-05 10:38:09 UTC] USER=www-data EUID=0 PID=827901 ACTION=fsop ARGS=chmod 700 /data/postgresql/17/zone-universe-main-dev/coordinator
[2026-02-05 10:38:09 UTC] USER=www-data EUID=0 PID=827922 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-zone-universe-main-dev-coordinator
[2026-02-05 10:38:09 UTC] USER=www-data EUID=0 PID=827943 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-zone-universe-main-dev-coordinator
[2026-02-05 10:38:09 UTC] USER=www-data EUID=0 PID=827974 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/zone-universe-main-dev/coordinator --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.73Riv1
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /data/postgresql/17/zone-universe-main-dev/coordinator ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/zone-universe-main-dev/coordinator -l logfile start

[OK]   initdb complete
[2026-02-05 10:38:11 UTC] USER=www-data EUID=0 PID=828092 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.73Riv1
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-02-05 10:38:11 UTC] USER=www-data EUID=0 PID=828153 ACTION=fsop ARGS=cp /tmp/tmp.R4M5Pzt5tK /data/postgresql/17/zone-universe-main-dev/coordinator/pg_hba.conf
[2026-02-05 10:38:11 UTC] USER=www-data EUID=0 PID=828175 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/zone-universe-main-dev/coordinator/pg_hba.conf
[2026-02-05 10:38:11 UTC] USER=www-data EUID=0 PID=828220 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/zone-universe-main-dev/coordinator/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@zone-universe-main-dev-coordinator.service
[2026-02-05 10:38:12 UTC] USER=www-data EUID=0 PID=828267 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@zone-universe-main-dev-coordinator.service
[OK]   systemd unit written
[2026-02-05 10:38:12 UTC] USER=www-data EUID=0 PID=828288 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 10:38:12 UTC] USER=www-data EUID=0 PID=828311 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 10:38:12 UTC] USER=www-data EUID=0 PID=828335 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-02-05 10:38:14 UTC] USER=www-data EUID=0 PID=828526 ACTION=passthru ARGS=systemctl start postgresql@zone-universe-main-dev-coordinator.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-02-05 10:38:14 UTC] USER=www-data EUID=0 PID=828591 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@zone-universe-main-dev-coordinator.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-zone-universe-main-dev-postgresql-coordinator.fastorder.com:5432)…
[WARN] TCP readiness via FQDN not confirmed (may be ACL-related)
[OK]   Startup sequence complete
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Provisioning application database and Debezium role (if not exists)...
[INFO] Checking if database fastorder_zone_universe_main_dev_db exists...
[INFO] DB check result: exit_code=0, output='[2026-02-05 10:42:20 UTC] USER=www-data EUID=0 PID=839363 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_zone_universe_main_dev_db''
[INFO] Creating database fastorder_zone_universe_main_dev_db...
[2026-02-05 10:42:20 UTC] USER=www-data EUID=0 PID=839389 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_zone_universe_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[OK]   Database fastorder_zone_universe_main_dev_db created
[INFO] Checking if role debezium_user exists...
[INFO] Role check result: exit_code=0, output='[2026-02-05 10:42:21 UTC] USER=www-data EUID=0 PID=839425 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[INFO] Creating role debezium_user...
[2026-02-05 10:42:21 UTC] USER=www-data EUID=0 PID=839458 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'le65Vqqbc8C4FWR0QRycEq6Q';
CREATE ROLE
[OK]   Role debezium_user created
[2026-02-05 10:42:22 UTC] USER=www-data EUID=0 PID=839499 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_zone_universe_main_dev_db" TO debezium_user;
GRANT
[OK]   Application DB (fastorder_zone_universe_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[INFO] Applying connection and memory optimizations...
[INFO] Current settings: max_connections=100, work_mem=4MB
[INFO] Target settings (coordinator): max_connections=150, work_mem=8MB
[2026-02-05 10:42:23 UTC] USER=www-data EUID=0 PID=839616 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 150;
ALTER SYSTEM
[2026-02-05 10:42:23 UTC] USER=www-data EUID=0 PID=839683 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-02-05 10:42:24 UTC] USER=www-data EUID=0 PID=839718 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   Settings applied to postgresql.auto.conf
[2026-02-05 10:42:24 UTC] USER=www-data EUID=0 PID=839733 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/zone-universe-main-dev/coordinator/standby.signal
[INFO] Service recently started (250s ago) - restarting to apply max_connections...
[INFO] Stopping service...
[2026-02-05 10:42:24 UTC] USER=www-data EUID=0 PID=839756 ACTION=passthru ARGS=systemctl stop postgresql@zone-universe-main-dev-coordinator.service
[INFO] Waiting for port 5432 to be released...
[OK]   Port 5432 released
[INFO] Starting service...
[2026-02-05 10:42:27 UTC] USER=www-data EUID=0 PID=839886 ACTION=passthru ARGS=systemctl start postgresql@zone-universe-main-dev-coordinator.service
[2026-02-05 10:42:33 UTC] USER=www-data EUID=0 PID=840210 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@zone-universe-main-dev-coordinator.service
[OK]   ✅ Optimization complete: max_connections=150, work_mem=8MB
[INFO] Setting postgres password via centralized script... for coordinator
[INFO] Temporarily disabling synchronous_commit on coordinator for password setting...
[OK]   Disabled synchronous_commit (was: on)
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    zone
  Zone:       universe
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/zone/universe/main/dev/postgresql/coordinator

Connection Info:
  Socket Dir: /var/run/postgresql-zone-universe-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Initial setup: Using password from initdb
✓ PostgreSQL password already set during initdb
Storing password in AWS Secrets Manager: fastorder/db/zone/universe/main/dev/postgresql/coordinator
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/zone/universe/main/dev/postgresql/coordinator
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/zone/universe/main/dev/postgresql/coordinator
✅ Secret created: fastorder/db/zone/universe/main/dev/postgresql/coordinator
✅ PostgreSQL credentials set in vault: fastorder/db/zone/universe/main/dev/postgresql/coordinator
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/zone/universe/main/dev/postgresql/coordinator
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
[INFO] Restoring synchronous_commit on coordinator...
[OK]   Restored synchronous_commit to: on
[OK]   Password set and persisted
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: zone-universe-main-dev
[INFO] Identifier: coordinator
[INFO] PostgreSQL IP: 10.100.1.55
[INFO] Primary hostname: db-zone-universe-main-dev-postgresql-coordinator.fastorder.com

[INFO] Adding /etc/hosts entries for coordinator...
[INFO]   1. db-zone-universe-main-dev-postgresql.fastorder.com → 10.100.1.55 (primary/short)
[INFO]   2. db-zone-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.55 (compatibility)

[INFO]   🔄 Updating db-zone-universe-main-dev-postgresql.fastorder.com → 10.100.1.55
✅     ✅ Updated: db-zone-universe-main-dev-postgresql.fastorder.com → 10.100.1.55
[INFO]   🔄 Updating db-zone-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.55
✅     ✅ Updated: db-zone-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.55

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.55    db-zone-universe-main-dev-postgresql.fastorder.com
  10.100.1.55    db-zone-universe-main-dev-postgresql-coordinator.fastorder.com


[OK]   PostgreSQL 'zone-universe-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.key \
        host=db-zone-universe-main-dev-postgresql-coordinator port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        zone-universe-main-dev-postgresql-coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.55
[INFO]   Port:              5432
[INFO]   FQDN:              db-zone-universe-main-dev-postgresql-coordinator
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 21addfb9-6a44-4dea-8573-7e2aa80db268
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 03 role...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[2026-02-05 10:42:47 UTC] USER=www-data EUID=0 PID=841201 ACTION=fsop ARGS=test -f /data/postgresql/17/zone-universe-main-dev/coordinator/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Username:    debezium_user
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   debezium_user
  Hostname:    db-zone-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:42:48 UTC] USER=www-data EUID=0 PID=841382 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-debezium_user
[2026-02-05 10:42:48 UTC] USER=www-data EUID=0 PID=841391 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-02-05 10:42:48 UTC] USER=www-data EUID=0 PID=841400 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-debezium_user/ra_root.key
[2026-02-05 10:42:48 UTC] USER=www-data EUID=0 PID=841409 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-02-05 10:42:48 UTC] USER=www-data EUID=0 PID=841423 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:42:49 UTC] USER=www-data EUID=0 PID=841461 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:42:49 UTC] USER=www-data EUID=0 PID=841470 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:42:49 UTC] USER=www-data EUID=0 PID=841479 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 10:42:49 UTC] USER=www-data EUID=0 PID=841490 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 10:42:49 UTC] USER=www-data EUID=0 PID=841514 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:42:49 UTC] USER=www-data EUID=0 PID=841525 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 10:42:49 UTC] USER=www-data EUID=0 PID=841534 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 10:42:49 UTC] USER=www-data EUID=0 PID=841543 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 10:42:49 UTC] USER=www-data EUID=0 PID=841552 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 10:42:49 UTC] USER=www-data EUID=0 PID=841561 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:42:49 UTC] USER=www-data EUID=0 PID=841570 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:42:49 UTC] USER=www-data EUID=0 PID=841579 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 10:42:49 UTC] USER=www-data EUID=0 PID=841588 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 10:42:49 UTC] USER=www-data EUID=0 PID=841597 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 10:42:49 UTC] USER=www-data EUID=0 PID=841607 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841619 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841663 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841672 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841681 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841690 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841702 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841711 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841721 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841730 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841740 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841752 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841761 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user_der.key /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841773 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user_pk8.der /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841783 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841794 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841806 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841817 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841831 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841844 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841861 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841870 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841879 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841888 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841897 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841906 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user_der.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841925 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user_pk8.der /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841937 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841947 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:42:50 UTC] USER=www-data EUID=0 PID=841956 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=841965 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=841974 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=841984 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=841993 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842002 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842011 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842020 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842029 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842038 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user_der.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842048 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user_pk8.der /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842058 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842068 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842077 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842086 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842095 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842104 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842113 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842122 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842131 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842144 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842155 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842164 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user_der.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842173 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/debezium_user_pk8.der /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842183 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842193 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842202 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842212 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842221 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842230 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842239 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842248 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842257 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:42:51 UTC] USER=www-data EUID=0 PID=842266 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: zone-universe-main-dev
User: debezium_user
Node: coordinator
FQDN: db-zone-universe-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/zone-universe-main-dev/coordinator/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-zone-universe-main-dev-postgresql-coordinator.fastorder.com -U debezium_user -d postgres

✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    zone
  Zone:       universe
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/zone/universe/main/dev/postgresql/coordinator/fastorder_admin_gd

Connection Info:
  Socket Dir: /var/run/postgresql-zone-universe-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: fastorder_admin_gd
Storing password in AWS Secrets Manager: fastorder/db/zone/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/zone/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/zone/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Secret created: fastorder/db/zone/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/zone/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/zone/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: coordinator/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-zone-universe-main-dev-postgresql.fastorder.com
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Username:    fastorder_admin_gd
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   fastorder_admin_gd
  Hostname:    db-zone-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:43:02 UTC] USER=www-data EUID=0 PID=843177 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-02-05 10:43:02 UTC] USER=www-data EUID=0 PID=843186 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
[2026-02-05 10:43:03 UTC] USER=www-data EUID=0 PID=843196 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-02-05 10:43:03 UTC] USER=www-data EUID=0 PID=843208 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:43:03 UTC] USER=www-data EUID=0 PID=843241 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:43:03 UTC] USER=www-data EUID=0 PID=843259 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 10:43:03 UTC] USER=www-data EUID=0 PID=843286 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 10:43:03 UTC] USER=www-data EUID=0 PID=843304 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:43:04 UTC] USER=www-data EUID=0 PID=843313 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 10:43:04 UTC] USER=www-data EUID=0 PID=843323 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 10:43:04 UTC] USER=www-data EUID=0 PID=843350 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 10:43:04 UTC] USER=www-data EUID=0 PID=843359 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 10:43:04 UTC] USER=www-data EUID=0 PID=843368 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 10:43:04 UTC] USER=www-data EUID=0 PID=843386 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[2026-02-05 10:43:04 UTC] USER=www-data EUID=0 PID=843407 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 10:43:04 UTC] USER=www-data EUID=0 PID=843416 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 10:43:04 UTC] USER=www-data EUID=0 PID=843425 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 10:43:04 UTC] USER=www-data EUID=0 PID=843434 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:43:04 UTC] USER=www-data EUID=0 PID=843443 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:43:04 UTC] USER=www-data EUID=0 PID=843490 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:43:04 UTC] USER=www-data EUID=0 PID=843499 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:43:05 UTC] USER=www-data EUID=0 PID=843521 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:43:05 UTC] USER=www-data EUID=0 PID=843533 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:43:05 UTC] USER=www-data EUID=0 PID=843551 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:43:05 UTC] USER=www-data EUID=0 PID=843572 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 10:43:05 UTC] USER=www-data EUID=0 PID=843581 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:43:05 UTC] USER=www-data EUID=0 PID=843590 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:43:05 UTC] USER=www-data EUID=0 PID=843599 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 10:43:05 UTC] USER=www-data EUID=0 PID=843608 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 10:43:05 UTC] USER=www-data EUID=0 PID=843617 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 10:43:05 UTC] USER=www-data EUID=0 PID=843630 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/zone-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:43:05 UTC] USER=www-data EUID=0 PID=843651 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:43:06 UTC] USER=www-data EUID=0 PID=843660 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:43:06 UTC] USER=www-data EUID=0 PID=843669 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:43:06 UTC] USER=www-data EUID=0 PID=843678 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:43:06 UTC] USER=www-data EUID=0 PID=843687 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 10:43:06 UTC] USER=www-data EUID=0 PID=843696 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 10:43:06 UTC] USER=www-data EUID=0 PID=843705 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:43:06 UTC] USER=www-data EUID=0 PID=843714 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:43:06 UTC] USER=www-data EUID=0 PID=843723 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 10:43:06 UTC] USER=www-data EUID=0 PID=843735 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 10:43:06 UTC] USER=www-data EUID=0 PID=843750 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 10:43:06 UTC] USER=www-data EUID=0 PID=843760 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:43:06 UTC] USER=www-data EUID=0 PID=843789 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:43:06 UTC] USER=www-data EUID=0 PID=843798 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:43:06 UTC] USER=www-data EUID=0 PID=843807 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:43:06 UTC] USER=www-data EUID=0 PID=843817 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:43:07 UTC] USER=www-data EUID=0 PID=843837 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 10:43:07 UTC] USER=www-data EUID=0 PID=843858 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:43:07 UTC] USER=www-data EUID=0 PID=843890 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:43:07 UTC] USER=www-data EUID=0 PID=843913 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 10:43:07 UTC] USER=www-data EUID=0 PID=843922 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 10:43:07 UTC] USER=www-data EUID=0 PID=843931 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 10:43:07 UTC] USER=www-data EUID=0 PID=843943 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:43:07 UTC] USER=www-data EUID=0 PID=843954 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:43:07 UTC] USER=www-data EUID=0 PID=843965 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:43:07 UTC] USER=www-data EUID=0 PID=843974 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:43:07 UTC] USER=www-data EUID=0 PID=843995 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator
[2026-02-05 10:43:07 UTC] USER=www-data EUID=0 PID=844015 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 10:43:07 UTC] USER=www-data EUID=0 PID=844029 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 10:43:07 UTC] USER=www-data EUID=0 PID=844044 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt
[2026-02-05 10:43:07 UTC] USER=www-data EUID=0 PID=844059 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
[2026-02-05 10:43:07 UTC] USER=www-data EUID=0 PID=844077 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 10:43:08 UTC] USER=www-data EUID=0 PID=844096 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 10:43:08 UTC] USER=www-data EUID=0 PID=844107 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:43:08 UTC] USER=www-data EUID=0 PID=844119 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:43:08 UTC] USER=www-data EUID=0 PID=844128 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:43:08 UTC] USER=www-data EUID=0 PID=844137 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 10:43:08 UTC] USER=www-data EUID=0 PID=844146 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 10:43:08 UTC] USER=www-data EUID=0 PID=844156 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 10:43:08 UTC] USER=www-data EUID=0 PID=844165 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:43:08 UTC] USER=www-data EUID=0 PID=844174 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:43:08 UTC] USER=www-data EUID=0 PID=844184 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:43:08 UTC] USER=www-data EUID=0 PID=844198 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: zone-universe-main-dev
User: fastorder_admin_gd
Node: coordinator
FQDN: db-zone-universe-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/zone-universe-main-dev/coordinator/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-zone-universe-main-dev-postgresql-coordinator.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-zone-universe-main-dev-coordinator:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_zone_universe_main_dev_db already exists, skipping creation
[2026-02-05 10:43:09 UTC] USER=www-data EUID=0 PID=844276 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-zone-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /data/postgresql/17/zone-universe-main-dev/coordinator/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-02-05 10:43:09 UTC] USER=www-data EUID=0 PID=844316 ACTION=passthru ARGS=systemctl reload postgresql@zone-universe-main-dev-coordinator.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

=== Pre-flight Checks ===
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
✓ AWS Secrets Manager accessible

=== Retrieving Credentials from AWS ===
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/zone/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/zone/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/zone/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials loaded for coordinator/fastorder_admin_gd: fastorder_admin_gd@db-zone-universe-main-dev-postgresql.fastorder.com:5432/fastorder_zone_universe_main_dev_db
✓ Credentials retrieved: fastorder_admin_gd@db-zone-universe-main-dev-postgresql.fastorder.com:5432/fastorder_zone_universe_main_dev_db
╔════════════════════════════════════════════╗
║  PostgreSQL Test Suite (AWS Secrets MGR)  ║
╚════════════════════════════════════════════╝

=== PostgreSQL Authentication Test ===
✗ PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-zone-universe-main-dev-postgresql.fastorder.com" (10.100.1.55), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/zone/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[2026-02-05 10:43:15 UTC] USER=www-data EUID=0 PID=844678 ACTION=fsop ARGS=test -f /data/postgresql/17/zone-universe-main-dev/coordinator/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : zone-universe-main-dev
  IDENTIFIER  : coordinator
  PG HOST     : db-zone-universe-main-dev-postgresql.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_zone_universe_main_dev_db
  SCHEMA      : zone
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator
  DNS → 10.100.1.55
  CA         : /home/www-data/ssl/.postgresql/zone-universe-main-dev/coordinator/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    zone
  Zone:       universe
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/zone/universe/main/dev/postgresql/coordinator/debezium_user

Connection Info:
  Socket Dir: /var/run/postgresql-zone-universe-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User debezium_user does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: debezium_user
Storing password in AWS Secrets Manager: fastorder/db/zone/universe/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/zone/universe/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/zone/universe/main/dev/postgresql/coordinator/debezium_user
✅ Secret created: fastorder/db/zone/universe/main/dev/postgresql/coordinator/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/zone/universe/main/dev/postgresql/coordinator/debezium_user
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/zone/universe/main/dev/postgresql/coordinator/debezium_user
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: coordinator/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-02-05 10:43:24 UTC] USER=www-data EUID=0 PID=845275 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-zone-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_zone_universe_main_dev_db already exists
[2026-02-05 10:43:25 UTC] USER=www-data EUID=0 PID=845320 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-zone-universe-main-dev-coordinator -p 5432 -d fastorder_zone_universe_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/zone-universe-main-dev/coordinator/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.

[DEBUG] Tracking substep start: steps/01-install/steps/04-install-extensions (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 04 install extensions...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

╔══════════════════════════════════════════════════════════════╗
║       PostgreSQL Extensions Installation                     ║
╠══════════════════════════════════════════════════════════════╣
║  Environment:  zone-universe-main-dev
║  Identifier:   coordinator
║  PG Version:   17
║  PostGIS:      yes
║  Citus:        yes
╚══════════════════════════════════════════════════════════════╝

[INFO] Installing PostgreSQL extension packages via wrapper...
[INFO] Updating package lists...
[2026-02-05 10:43:27 UTC] USER=www-data EUID=0 PID=845426 ACTION=passthru ARGS=apt-get update -qq
[WARN] apt-get update had warnings (continuing)
[INFO] Installing postgresql-17-pgvector...
[2026-02-05 10:43:27 UTC] USER=www-data EUID=0 PID=845456 ACTION=passthru ARGS=apt-get install -y -qq postgresql-17-pgvector
[WARN] Failed to install postgresql-17-pgvector (may not be available for PG 17)
[OK] Package postgresql-17-postgis-3 already installed
[OK] Package postgresql-17-postgis-3-scripts already installed
[INFO] Database connection:
[INFO]   Host: db-zone-universe-main-dev-postgresql.fastorder.com:5432
[INFO]   Database: fastorder_zone_universe_main_dev_db
[INFO]   SSL Dir: /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[INFO] Checking PostgreSQL connectivity...
[OK] PostgreSQL is accessible
[INFO] Installing extensions in postgres database...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Installing Extensions in: postgres
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Installing hstore in postgres...
CREATE EXTENSION
[OK] Installed hstore in postgres
[INFO] Installing vector in postgres...
[WARN] Failed to install vector in postgres (may require superuser or package)
[INFO] Installing ltree in postgres...
CREATE EXTENSION
[OK] Installed ltree in postgres
[INFO] Installing dblink in postgres...
CREATE EXTENSION
[OK] Installed dblink in postgres
[INFO] Installing fuzzystrmatch in postgres...
CREATE EXTENSION
[OK] Installed fuzzystrmatch in postgres
[INFO] Installing pgcrypto in postgres...
CREATE EXTENSION
[OK] Installed pgcrypto in postgres
[INFO] Installing btree_gist in postgres...
CREATE EXTENSION
[OK] Installed btree_gist in postgres
[INFO] Installing pg_trgm in postgres...
CREATE EXTENSION
[OK] Installed pg_trgm in postgres
[INFO] Installing btree_gin in postgres...
CREATE EXTENSION
[OK] Installed btree_gin in postgres
[INFO] Installing uuid-ossp in postgres...
CREATE EXTENSION
[OK] Installed uuid-ossp in postgres
[INFO] Installing unaccent in postgres...
CREATE EXTENSION
[OK] Installed unaccent in postgres
[INFO] Installing PostGIS extensions...
[INFO] Installing postgis in postgres...
CREATE EXTENSION
[OK] Installed postgis in postgres
[INFO] Installing postgis_topology in postgres...
CREATE EXTENSION
[OK] Installed postgis_topology in postgres
[INFO] Installing Citus extension (coordinator)...
[INFO] Installing citus in postgres...
CREATE EXTENSION
[OK] Installed citus in postgres
[INFO] Installing extensions in fastorder_zone_universe_main_dev_db...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Installing Extensions in: fastorder_zone_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Installing hstore in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed hstore in fastorder_zone_universe_main_dev_db
[INFO] Installing vector in fastorder_zone_universe_main_dev_db...
[WARN] Failed to install vector in fastorder_zone_universe_main_dev_db (may require superuser or package)
[INFO] Installing ltree in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed ltree in fastorder_zone_universe_main_dev_db
[INFO] Installing dblink in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed dblink in fastorder_zone_universe_main_dev_db
[INFO] Installing fuzzystrmatch in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed fuzzystrmatch in fastorder_zone_universe_main_dev_db
[INFO] Installing pgcrypto in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed pgcrypto in fastorder_zone_universe_main_dev_db
[INFO] Installing btree_gist in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed btree_gist in fastorder_zone_universe_main_dev_db
[INFO] Installing pg_trgm in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed pg_trgm in fastorder_zone_universe_main_dev_db
[INFO] Installing btree_gin in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed btree_gin in fastorder_zone_universe_main_dev_db
[INFO] Installing uuid-ossp in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed uuid-ossp in fastorder_zone_universe_main_dev_db
[INFO] Installing unaccent in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed unaccent in fastorder_zone_universe_main_dev_db
[INFO] Installing PostGIS extensions...
[INFO] Installing postgis in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed postgis in fastorder_zone_universe_main_dev_db
[INFO] Installing postgis_topology in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed postgis_topology in fastorder_zone_universe_main_dev_db
[INFO] Installing Citus extension (coordinator)...
[INFO] Installing citus in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed citus in fastorder_zone_universe_main_dev_db

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Extension Verification: postgres
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ✓ btree_gin        | 1.3
  ✓ btree_gist       | 1.7
  ✓ citus            | 13.2-1
  ✓ dblink           | 1.2
  ✓ fuzzystrmatch    | 1.2
  ✓ hstore           | 1.8
  ✓ ltree            | 1.3
  ✓ pg_trgm          | 1.6
  ✓ pgcrypto         | 1.3
  ✓ plpgsql          | 1.0
  ✓ postgis          | 3.5.3
  ✓ postgis_topology | 3.5.3
  ✓ unaccent         | 1.1
  ✓ uuid-ossp        | 1.1


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Extension Verification: fastorder_zone_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ✓ btree_gin        | 1.3
  ✓ btree_gist       | 1.7
  ✓ citus            | 13.2-1
  ✓ dblink           | 1.2
  ✓ fuzzystrmatch    | 1.2
  ✓ hstore           | 1.8
  ✓ ltree            | 1.3
  ✓ pg_trgm          | 1.6
  ✓ pgcrypto         | 1.3
  ✓ plpgsql          | 1.0
  ✓ postgis          | 3.5.3
  ✓ postgis_topology | 3.5.3
  ✓ unaccent         | 1.1
  ✓ uuid-ossp        | 1.1


╔══════════════════════════════════════════════════════════════╗
║  ✅ PostgreSQL Extensions Installation Complete              ║
╚══════════════════════════════════════════════════════════════╝


[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (zone) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=zone
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/zone/run.sh
🔍 DEBUG_CHECKPOINT_03: No specific folder for zone, using default
[DEBUG] Tracking substep start: steps/01-install/steps/default (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 🔸 Service: zone (using default contracts schema)
🔍 DEBUG_CHECKPOINT_04: Executing default: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/run.sh
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] 🟢 Starting default contracts schema provisioning for SERVICE=zone
[INFO] Environment: zone-universe-main-dev
[INFO] Schema: zone (contracts tables)
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG: Looking for contracts steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/default/contracts/steps
[INFO] 📁 Running contracts schema setup for: zone
[INFO] 📁 Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/default/contracts/steps

[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing zone schema (contracts tables)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Schema:      zone
  Identifier:  coordinator
  Database:    fastorder_zone_universe_main_dev_db
  Host:        db-zone-universe-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_zone_universe_main_dev_db
ℹ️  Database fastorder_zone_universe_main_dev_db already exists
✅ Connected to database: fastorder_zone_universe_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating zone schema...
NOTICE:  schema "zone" already exists, skipping
CREATE SCHEMA
✅ Schema created
🔧 Creating contracts tables in zone schema...
   Creating "zone".contract_key...
CREATE TABLE
   Creating "zone".contract_type...
CREATE TABLE
   Creating "zone".contracts...
CREATE TABLE
   Adding columns to "zone".contracts (safe migration)...
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
UPDATE 0
UPDATE 0
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
   Creating "zone".contract_vars...
CREATE TABLE
   Creating "zone".contract_datetime...
CREATE TABLE
   Creating "zone".contract_decimal...
CREATE TABLE
   Creating "zone".contract_float...
CREATE TABLE
   Creating "zone".contract_int...
CREATE TABLE
   Creating "zone".contract_json...
CREATE TABLE
   Creating "zone".contract_terms...
CREATE TABLE
   Creating "zone".contract_term_contracts...
CREATE TABLE
   Creating "zone".contract_term_datetime...
CREATE TABLE
   Creating "zone".contract_term_decimal...
CREATE TABLE
   Creating "zone".contract_term_float...
CREATE TABLE
   Creating "zone".contract_term_int...
CREATE TABLE
   Creating "zone".contract_term_items...
CREATE TABLE
   Creating "zone".contract_term_json...
CREATE TABLE
   Creating "zone".contract_term_vars...
CREATE TABLE
   Creating "zone".zone_id_uuid_mapping...
CREATE TABLE
✅ All 19 tables created
🔧 Creating indexes...
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
✅ All indexes created
🔧 Creating foreign keys...
DO
DO
✅ Foreign keys created
🔧 Configuring Citus distribution...
   Creating reference table: contract_key
 create_reference_table 
------------------------
 
(1 row)

   Creating reference table: contract_type
 create_reference_table 
------------------------
 
(1 row)

   Creating distributed table: contracts
   Creating distributed table: contract_vars
   Creating distributed table: contract_datetime
   Creating distributed table: contract_decimal
   Creating distributed table: contract_float
   Creating distributed table: contract_int
   Creating distributed table: contract_json
   Creating distributed table: contract_terms
   Creating distributed table: contract_term_contracts
   Creating distributed table: contract_term_datetime
   Creating distributed table: contract_term_decimal
   Creating distributed table: contract_term_float
   Creating distributed table: contract_term_int
   Creating distributed table: contract_term_items
   Creating distributed table: contract_term_json
 create_distributed_table 
--------------------------
 
(1 row)

   Creating distributed table: contract_term_vars
 create_distributed_table 
--------------------------
 
(1 row)

✅ Citus distribution configured
🎉 Schema initialization complete for zone in fastorder_zone_universe_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

==========================================
✅ zone schema initialization complete!
   Tables: 19
   Indexes: 54
==========================================

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Default contracts schema setup complete for: zone
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

✓ ✅ Coordinator setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up 1 worker(s) (Citus data nodes)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up worker: worker-01
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-02-05 10:43:57 UTC] USER=unknown EUID=33 PID=847460 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 10:43:57 UTC] USER=unknown EUID=33 PID=847467 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 10:43:57 UTC] USER=unknown EUID=33 PID=847475 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-02-05 10:43:57 UTC] USER=unknown EUID=33 PID=847483 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 10:43:57 UTC] USER=unknown EUID=33 PID=847491 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for zone in universe-dev...
[INFO] Environment: zone-universe-main-dev
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172 JOB_UUID=20cfe3cc-b016-4175-bac8-362f4003c72d

[2026-02-05 10:43:57 UTC] USER=unknown EUID=33 PID=847499 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[WARN] Could not find PostgreSQL IP for worker-01 in topology.json, allocating new VM IP...
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Allocated new VM IP: 10.100.1.106 for db-worker-01-postgresql
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: zone-universe-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.106
[INFO] Primary hostname: db-zone-universe-main-dev-postgresql-worker-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01...
[INFO]   db-zone-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.106

[INFO]   ➕ Adding db-zone-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.106
✅     ✅ Added: db-zone-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.106

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.106    db-zone-universe-main-dev-postgresql-worker-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      zone-universe-main-dev.fastorder.com
  Internal IP: 10.100.1.106 (from topology: db-worker-01-postgresql)
  Shared IP:   142.93.238.16 (legacy/public, also included in SANs)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 10:44:02 UTC] USER=www-data EUID=0 PID=847958 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01 and /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:44:02 UTC] USER=www-data EUID=0 PID=847972 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-02-05 10:44:02 UTC] USER=www-data EUID=0 PID=848013 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-847896
[2026-02-05 10:44:02 UTC] USER=www-data EUID=0 PID=848076 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-847896/ra_root.crt
[2026-02-05 10:44:02 UTC] USER=www-data EUID=0 PID=848104 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-847896/ra_root.key
[2026-02-05 10:44:02 UTC] USER=www-data EUID=0 PID=848126 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-847896/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
[2026-02-05 10:44:04 UTC] USER=www-data EUID=0 PID=848171 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-847896/server.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.key
[2026-02-05 10:44:04 UTC] USER=www-data EUID=0 PID=848180 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-847896/server.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.crt
[2026-02-05 10:44:04 UTC] USER=www-data EUID=0 PID=848190 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-02-05 10:44:04 UTC] USER=www-data EUID=0 PID=848211 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:44:04 UTC] USER=www-data EUID=0 PID=848237 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 10:44:04 UTC] USER=www-data EUID=0 PID=848248 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.key
[2026-02-05 10:44:04 UTC] USER=www-data EUID=0 PID=848257 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.key
[2026-02-05 10:44:04 UTC] USER=www-data EUID=0 PID=848266 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.crt
[2026-02-05 10:44:04 UTC] USER=www-data EUID=0 PID=848275 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.crt
[2026-02-05 10:44:04 UTC] USER=www-data EUID=0 PID=848285 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:44:04 UTC] USER=www-data EUID=0 PID=848294 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-zone-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:zone-universe-main-dev.fastorder.com, DNS:db-zone-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:db-zone-universe-main-dev-postgresql-worker-01, DNS:localhost, IP Address:10.100.1.106, IP Address:127.0.0.1, IP Address:142.93.238.16
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: zone-universe-main-dev
Node:        worker-01
Primary CN:  db-zone-universe-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@zone-universe-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-zone-universe-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:44:05 UTC] USER=www-data EUID=0 PID=848424 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-02-05 10:44:05 UTC] USER=www-data EUID=0 PID=848435 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 10:44:05 UTC] USER=www-data EUID=0 PID=848444 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-02-05 10:44:05 UTC] USER=www-data EUID=0 PID=848454 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 10:44:05 UTC] USER=www-data EUID=0 PID=848463 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:44:06 UTC] USER=www-data EUID=0 PID=848488 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:44:06 UTC] USER=www-data EUID=0 PID=848500 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:44:06 UTC] USER=www-data EUID=0 PID=848509 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key
[2026-02-05 10:44:06 UTC] USER=www-data EUID=0 PID=848518 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.crt
[2026-02-05 10:44:06 UTC] USER=www-data EUID=0 PID=848527 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:44:06 UTC] USER=www-data EUID=0 PID=848536 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:44:06 UTC] USER=www-data EUID=0 PID=848548 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 10:44:06 UTC] USER=www-data EUID=0 PID=848561 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 10:44:06 UTC] USER=www-data EUID=0 PID=848572 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 10:44:06 UTC] USER=www-data EUID=0 PID=848596 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key
[2026-02-05 10:44:06 UTC] USER=www-data EUID=0 PID=848605 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:44:06 UTC] USER=www-data EUID=0 PID=848620 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:44:06 UTC] USER=www-data EUID=0 PID=848630 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key
[2026-02-05 10:44:06 UTC] USER=www-data EUID=0 PID=848640 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 10:44:06 UTC] USER=www-data EUID=0 PID=848658 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 10:44:06 UTC] USER=www-data EUID=0 PID=848671 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:44:07 UTC] USER=www-data EUID=0 PID=848680 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:07 UTC] USER=www-data EUID=0 PID=848719 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:07 UTC] USER=www-data EUID=0 PID=848728 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:44:07 UTC] USER=www-data EUID=0 PID=848737 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:44:07 UTC] USER=www-data EUID=0 PID=848767 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:07 UTC] USER=www-data EUID=0 PID=848782 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key
[2026-02-05 10:44:07 UTC] USER=www-data EUID=0 PID=848792 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=848803 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=848816 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=848828 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=848853 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_pk8.der /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=848872 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=848884 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=848893 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=848902 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=848912 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=848921 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=848944 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=848959 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=848968 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=848977 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=848986 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=848995 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=849004 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_pk8.der /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=849014 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:08 UTC] USER=www-data EUID=0 PID=849024 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:09 UTC] USER=www-data EUID=0 PID=849033 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:44:09 UTC] USER=www-data EUID=0 PID=849042 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:44:09 UTC] USER=www-data EUID=0 PID=849051 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:09 UTC] USER=www-data EUID=0 PID=849060 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:09 UTC] USER=www-data EUID=0 PID=849078 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt
[2026-02-05 10:44:09 UTC] USER=www-data EUID=0 PID=849087 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:44:09 UTC] USER=www-data EUID=0 PID=849096 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:44:09 UTC] USER=www-data EUID=0 PID=849106 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 10:44:09 UTC] USER=www-data EUID=0 PID=849117 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 10:44:09 UTC] USER=www-data EUID=0 PID=849126 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 10:44:09 UTC] USER=www-data EUID=0 PID=849136 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:09 UTC] USER=www-data EUID=0 PID=849146 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:09 UTC] USER=www-data EUID=0 PID=849155 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:44:09 UTC] USER=www-data EUID=0 PID=849165 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:44:09 UTC] USER=www-data EUID=0 PID=849175 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:09 UTC] USER=www-data EUID=0 PID=849187 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:09 UTC] USER=www-data EUID=0 PID=849198 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key
[2026-02-05 10:44:10 UTC] USER=www-data EUID=0 PID=849218 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:44:10 UTC] USER=www-data EUID=0 PID=849227 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:44:10 UTC] USER=www-data EUID=0 PID=849236 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 10:44:10 UTC] USER=www-data EUID=0 PID=849252 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 10:44:10 UTC] USER=www-data EUID=0 PID=849288 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:44:10 UTC] USER=www-data EUID=0 PID=849305 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:44:10 UTC] USER=www-data EUID=0 PID=849314 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:44:10 UTC] USER=www-data EUID=0 PID=849323 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 10:44:10 UTC] USER=www-data EUID=0 PID=849332 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 10:44:10 UTC] USER=www-data EUID=0 PID=849341 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 10:44:10 UTC] USER=www-data EUID=0 PID=849351 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:44:10 UTC] USER=www-data EUID=0 PID=849360 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:44:10 UTC] USER=www-data EUID=0 PID=849369 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:44:10 UTC] USER=www-data EUID=0 PID=849378 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: zone-universe-main-dev
User: postgres
Node: worker-01
FQDN: db-zone-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-zone-universe-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:44:11 UTC] USER=www-data EUID=0 PID=849440 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-02-05 10:44:11 UTC] USER=www-data EUID=0 PID=849449 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 10:44:11 UTC] USER=www-data EUID=0 PID=849458 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:44:11 UTC] USER=www-data EUID=0 PID=849507 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849516 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849525 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849534 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.crt
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849543 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849552 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849561 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849570 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849579 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849596 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849612 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849631 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849646 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849664 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849673 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849697 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849724 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849734 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:44:12 UTC] USER=www-data EUID=0 PID=849743 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:13 UTC] USER=www-data EUID=0 PID=849769 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:13 UTC] USER=www-data EUID=0 PID=849781 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:44:13 UTC] USER=www-data EUID=0 PID=849802 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:13 UTC] USER=www-data EUID=0 PID=849811 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:13 UTC] USER=www-data EUID=0 PID=849822 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key
[2026-02-05 10:44:13 UTC] USER=www-data EUID=0 PID=849843 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:44:13 UTC] USER=www-data EUID=0 PID=849854 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:44:13 UTC] USER=www-data EUID=0 PID=849867 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 10:44:13 UTC] USER=www-data EUID=0 PID=849878 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 10:44:13 UTC] USER=www-data EUID=0 PID=849888 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_pk8.der /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 10:44:13 UTC] USER=www-data EUID=0 PID=849898 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:13 UTC] USER=www-data EUID=0 PID=849908 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:13 UTC] USER=www-data EUID=0 PID=849918 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:44:13 UTC] USER=www-data EUID=0 PID=849927 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:44:13 UTC] USER=www-data EUID=0 PID=849936 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:13 UTC] USER=www-data EUID=0 PID=849945 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=849954 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=849963 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=849972 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=849981 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=849990 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=849999 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=850008 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_pk8.der /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=850018 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=850028 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=850037 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=850046 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=850055 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=850064 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=850073 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=850082 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=850091 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=850100 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=850109 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=850129 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=850147 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 10:44:14 UTC] USER=www-data EUID=0 PID=850159 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:15 UTC] USER=www-data EUID=0 PID=850213 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:44:15 UTC] USER=www-data EUID=0 PID=850224 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:15 UTC] USER=www-data EUID=0 PID=850233 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:44:15 UTC] USER=www-data EUID=0 PID=850242 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key
[2026-02-05 10:44:15 UTC] USER=www-data EUID=0 PID=850251 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt
[2026-02-05 10:44:15 UTC] USER=www-data EUID=0 PID=850269 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:44:15 UTC] USER=www-data EUID=0 PID=850278 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 10:44:15 UTC] USER=www-data EUID=0 PID=850288 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 10:44:15 UTC] USER=www-data EUID=0 PID=850300 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres_pk8.der /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 10:44:15 UTC] USER=www-data EUID=0 PID=850310 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:44:15 UTC] USER=www-data EUID=0 PID=850341 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:44:15 UTC] USER=www-data EUID=0 PID=850350 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 10:44:16 UTC] USER=www-data EUID=0 PID=850361 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
   ✅ Symlinked client-key.pem
[2026-02-05 10:44:16 UTC] USER=www-data EUID=0 PID=850383 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:44:16 UTC] USER=www-data EUID=0 PID=850392 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:44:16 UTC] USER=www-data EUID=0 PID=850401 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:44:16 UTC] USER=www-data EUID=0 PID=850410 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: zone-universe-main-dev
User: postgres
Node: worker-01
FQDN: db-zone-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-zone-universe-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-postgresql environment: db-zone-universe-main-dev-postgresql-worker-01.fastorder.com (10.100.1.106)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.106
[INFO] Environment: zone-universe-main-dev
[INFO] Identifier: worker-01
[INFO] Data dir:   /data/postgresql/17/zone-universe-main-dev/worker-01
[INFO] Port:       5432
[INFO] Hostname:   db-zone-universe-main-dev-postgresql-worker-01
[2026-02-05 10:44:18 UTC] USER=www-data EUID=0 PID=850574 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:44:18 UTC] USER=www-data EUID=0 PID=850597 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:44:18 UTC] USER=www-data EUID=0 PID=850618 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:44:18 UTC] USER=www-data EUID=0 PID=850641 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      zone-universe-main-dev.fastorder.com
  Internal IP: 10.100.1.106 (from topology: db-worker-01-postgresql)
  Shared IP:   142.93.238.16 (legacy/public, also included in SANs)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 10:44:18 UTC] USER=www-data EUID=0 PID=850709 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01 and /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:44:18 UTC] USER=www-data EUID=0 PID=850725 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-02-05 10:44:18 UTC] USER=www-data EUID=0 PID=850735 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-850648
[2026-02-05 10:44:18 UTC] USER=www-data EUID=0 PID=850744 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-850648/ra_root.crt
[2026-02-05 10:44:19 UTC] USER=www-data EUID=0 PID=850753 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-850648/ra_root.key
[2026-02-05 10:44:19 UTC] USER=www-data EUID=0 PID=850762 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-850648/ra_root.crt
[2026-02-05 10:44:19 UTC] USER=www-data EUID=0 PID=850771 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-850648/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
[2026-02-05 10:44:20 UTC] USER=www-data EUID=0 PID=850875 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-02-05 10:44:20 UTC] USER=www-data EUID=0 PID=850884 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-850648/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:44:20 UTC] USER=www-data EUID=0 PID=850893 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:44:20 UTC] USER=www-data EUID=0 PID=850902 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 10:44:20 UTC] USER=www-data EUID=0 PID=850926 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.key
[2026-02-05 10:44:21 UTC] USER=www-data EUID=0 PID=850936 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.key
[2026-02-05 10:44:21 UTC] USER=www-data EUID=0 PID=850952 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.crt
[2026-02-05 10:44:21 UTC] USER=www-data EUID=0 PID=850961 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.crt
[2026-02-05 10:44:21 UTC] USER=www-data EUID=0 PID=850970 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:44:21 UTC] USER=www-data EUID=0 PID=850979 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-zone-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:zone-universe-main-dev.fastorder.com, DNS:db-zone-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:db-zone-universe-main-dev-postgresql-worker-01, DNS:localhost, IP Address:10.100.1.106, IP Address:127.0.0.1, IP Address:142.93.238.16
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: zone-universe-main-dev
Node:        worker-01
Primary CN:  db-zone-universe-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@zone-universe-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-zone-universe-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-02-05 10:44:21 UTC] USER=www-data EUID=0 PID=851018 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.key
[2026-02-05 10:44:21 UTC] USER=www-data EUID=0 PID=851027 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-02-05 10:44:21 UTC] USER=www-data EUID=0 PID=851053 ACTION=passthru ARGS=systemctl stop postgresql@zone-universe-main-dev-worker-01.service
[2026-02-05 10:44:21 UTC] USER=www-data EUID=0 PID=851095 ACTION=passthru ARGS=systemctl stop postgresql
[OK]   No conflicting Postgres left on port 5432
[OK]   Generated new postgres password for initdb
[2026-02-05 10:44:46 UTC] USER=www-data EUID=0 PID=852243 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.XaxuXE
[2026-02-05 10:44:46 UTC] USER=www-data EUID=0 PID=852264 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.XaxuXE
[2026-02-05 10:44:47 UTC] USER=www-data EUID=0 PID=852289 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/zone-universe-main-dev
[2026-02-05 10:44:47 UTC] USER=www-data EUID=0 PID=852312 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/zone-universe-main-dev
[2026-02-05 10:44:47 UTC] USER=www-data EUID=0 PID=852335 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/zone-universe-main-dev
[INFO] Initializing cluster in /data/postgresql/17/zone-universe-main-dev/worker-01 (SCRAM; pwfile)
[2026-02-05 10:44:47 UTC] USER=www-data EUID=0 PID=852362 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/zone-universe-main-dev/worker-01
[2026-02-05 10:44:47 UTC] USER=www-data EUID=0 PID=852390 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/zone-universe-main-dev/worker-01
[2026-02-05 10:44:47 UTC] USER=www-data EUID=0 PID=852418 ACTION=fsop ARGS=chmod 700 /data/postgresql/17/zone-universe-main-dev/worker-01
[2026-02-05 10:44:47 UTC] USER=www-data EUID=0 PID=852441 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-zone-universe-main-dev-worker-01
[2026-02-05 10:44:47 UTC] USER=www-data EUID=0 PID=852478 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-zone-universe-main-dev-worker-01
[2026-02-05 10:44:47 UTC] USER=www-data EUID=0 PID=852499 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-zone-universe-main-dev-worker-01
[2026-02-05 10:44:47 UTC] USER=www-data EUID=0 PID=852512 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/zone-universe-main-dev/worker-01 --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.XaxuXE
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /data/postgresql/17/zone-universe-main-dev/worker-01 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/zone-universe-main-dev/worker-01 -l logfile start

[OK]   initdb complete
[2026-02-05 10:44:48 UTC] USER=www-data EUID=0 PID=852571 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.XaxuXE
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-02-05 10:44:48 UTC] USER=www-data EUID=0 PID=852626 ACTION=fsop ARGS=cp /tmp/tmp.wILlIXV1kG /data/postgresql/17/zone-universe-main-dev/worker-01/pg_hba.conf
[2026-02-05 10:44:49 UTC] USER=www-data EUID=0 PID=852651 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/zone-universe-main-dev/worker-01/pg_hba.conf
[2026-02-05 10:44:49 UTC] USER=www-data EUID=0 PID=852676 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/zone-universe-main-dev/worker-01/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@zone-universe-main-dev-worker-01.service
[2026-02-05 10:44:49 UTC] USER=www-data EUID=0 PID=852710 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.2qrMPA /etc/systemd/system/postgresql@zone-universe-main-dev-worker-01.service
[2026-02-05 10:44:49 UTC] USER=www-data EUID=0 PID=852731 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@zone-universe-main-dev-worker-01.service
[OK]   systemd unit written
[2026-02-05 10:44:49 UTC] USER=www-data EUID=0 PID=852752 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 10:44:49 UTC] USER=www-data EUID=0 PID=852774 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 10:44:49 UTC] USER=www-data EUID=0 PID=852795 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-02-05 10:44:50 UTC] USER=www-data EUID=0 PID=852934 ACTION=passthru ARGS=systemctl start postgresql@zone-universe-main-dev-worker-01.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-02-05 10:44:51 UTC] USER=www-data EUID=0 PID=853021 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@zone-universe-main-dev-worker-01.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-zone-universe-main-dev-postgresql-worker-01.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Provisioning application database and Debezium role (if not exists)...
[INFO] Checking if database fastorder_zone_universe_main_dev_db exists...
[INFO] DB check result: exit_code=0, output='[2026-02-05 10:44:52 UTC] USER=www-data EUID=0 PID=853218 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_zone_universe_main_dev_db''
[INFO] Creating database fastorder_zone_universe_main_dev_db...
[2026-02-05 10:44:52 UTC] USER=www-data EUID=0 PID=853244 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_zone_universe_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[OK]   Database fastorder_zone_universe_main_dev_db created
[INFO] Checking if role debezium_user exists...
[INFO] Role check result: exit_code=0, output='[2026-02-05 10:44:52 UTC] USER=www-data EUID=0 PID=853332 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[INFO] Creating role debezium_user...
[2026-02-05 10:44:53 UTC] USER=www-data EUID=0 PID=853366 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'WBZIuaXVgtnSmHONS7HId3p2';
CREATE ROLE
[OK]   Role debezium_user created
[2026-02-05 10:44:53 UTC] USER=www-data EUID=0 PID=853395 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_zone_universe_main_dev_db" TO debezium_user;
GRANT
[OK]   Application DB (fastorder_zone_universe_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[INFO] Applying connection and memory optimizations...
[INFO] Current settings: max_connections=100, work_mem=4MB
[INFO] Target settings (worker): max_connections=100, work_mem=8MB
[2026-02-05 10:44:54 UTC] USER=www-data EUID=0 PID=853495 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 100;
ALTER SYSTEM
[2026-02-05 10:44:54 UTC] USER=www-data EUID=0 PID=853527 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-02-05 10:44:54 UTC] USER=www-data EUID=0 PID=853566 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   Settings applied to postgresql.auto.conf
[2026-02-05 10:44:54 UTC] USER=www-data EUID=0 PID=853602 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/zone-universe-main-dev/worker-01/standby.signal
[INFO] Service recently started (3s ago) - restarting to apply max_connections...
[INFO] Stopping service...
[2026-02-05 10:44:55 UTC] USER=www-data EUID=0 PID=853635 ACTION=passthru ARGS=systemctl stop postgresql@zone-universe-main-dev-worker-01.service
[INFO] Waiting for port 5432 to be released...
[OK]   Port 5432 released
[INFO] Starting service...
[2026-02-05 10:44:58 UTC] USER=www-data EUID=0 PID=853863 ACTION=passthru ARGS=systemctl start postgresql@zone-universe-main-dev-worker-01.service
[2026-02-05 10:45:08 UTC] USER=www-data EUID=0 PID=855620 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@zone-universe-main-dev-worker-01.service
[OK]   ✅ Optimization complete: max_connections=100, work_mem=8MB
[OK]   Synchronous replication already configured (synchronous_commit: on)
[INFO] Setting postgres password via centralized script... for worker-01
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    zone
  Zone:       universe
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/zone/universe/main/dev/postgresql/worker-01

Connection Info:
  Socket Dir: /var/run/postgresql-zone-universe-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Initial setup: Using password from initdb
✓ PostgreSQL password already set during initdb
Storing password in AWS Secrets Manager: fastorder/db/zone/universe/main/dev/postgresql/worker-01
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/zone/universe/main/dev/postgresql/worker-01
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/zone/universe/main/dev/postgresql/worker-01
✅ Secret created: fastorder/db/zone/universe/main/dev/postgresql/worker-01
✅ PostgreSQL credentials set in vault: fastorder/db/zone/universe/main/dev/postgresql/worker-01
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/zone/universe/main/dev/postgresql/worker-01
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
[OK]   Password set and persisted
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: zone-universe-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.106
[INFO] Primary hostname: db-zone-universe-main-dev-postgresql-worker-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01...
[INFO]   db-zone-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.106

[INFO]   ✅ db-zone-universe-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.106    db-zone-universe-main-dev-postgresql-worker-01.fastorder.com


[OK]   PostgreSQL 'zone-universe-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.key \
        host=db-zone-universe-main-dev-postgresql-worker-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        zone-universe-main-dev-postgresql-worker-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.106
[INFO]   Port:              5432
[INFO]   FQDN:              db-zone-universe-main-dev-postgresql-worker-01
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: a3189020-6bec-4e6c-9249-7f6d6b4de0f1
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 03 role...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[2026-02-05 10:45:40 UTC] USER=www-data EUID=0 PID=861173 ACTION=fsop ARGS=test -f /data/postgresql/17/zone-universe-main-dev/worker-01/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Username:    debezium_user
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   debezium_user
  Hostname:    db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:45:41 UTC] USER=www-data EUID=0 PID=861376 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-debezium_user
[2026-02-05 10:45:41 UTC] USER=www-data EUID=0 PID=861388 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-02-05 10:45:41 UTC] USER=www-data EUID=0 PID=861401 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-debezium_user/ra_root.key
[2026-02-05 10:45:41 UTC] USER=www-data EUID=0 PID=861411 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-02-05 10:45:41 UTC] USER=www-data EUID=0 PID=861420 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:45:42 UTC] USER=www-data EUID=0 PID=861437 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:45:42 UTC] USER=www-data EUID=0 PID=861447 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:45:42 UTC] USER=www-data EUID=0 PID=861456 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 10:45:42 UTC] USER=www-data EUID=0 PID=861465 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 10:45:42 UTC] USER=www-data EUID=0 PID=861474 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:45:42 UTC] USER=www-data EUID=0 PID=861483 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:45:42 UTC] USER=www-data EUID=0 PID=861502 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 10:45:42 UTC] USER=www-data EUID=0 PID=861511 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 10:45:42 UTC] USER=www-data EUID=0 PID=861520 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 10:45:42 UTC] USER=www-data EUID=0 PID=861529 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:45:42 UTC] USER=www-data EUID=0 PID=861540 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:45:42 UTC] USER=www-data EUID=0 PID=861549 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 10:45:42 UTC] USER=www-data EUID=0 PID=861561 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861570 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861579 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861588 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861597 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861625 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861634 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861653 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861676 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861685 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861694 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861703 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861712 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861721 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user_der.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861730 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user_pk8.der /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861740 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861750 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861760 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861769 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861778 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861787 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:43 UTC] USER=www-data EUID=0 PID=861796 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=861808 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=861818 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=861827 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=861836 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=861845 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user_der.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=861854 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user_pk8.der /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=861864 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=861874 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=861895 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=861905 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=861920 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=861931 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=861941 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=861953 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=861963 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=861981 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=862009 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=862020 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user_der.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=862029 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user_pk8.der /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=862042 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:44 UTC] USER=www-data EUID=0 PID=862052 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862061 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862070 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862079 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862088 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862097 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862106 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862115 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862124 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862133 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862142 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user_der.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862151 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/debezium_user_pk8.der /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862161 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862171 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862191 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862200 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
   ✅ Symlinked client-key.pem
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862218 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862227 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:45:45 UTC] USER=www-data EUID=0 PID=862245 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: zone-universe-main-dev
User: debezium_user
Node: worker-01
FQDN: db-zone-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-zone-universe-main-dev-postgresql-worker-01.fastorder.com -U debezium_user -d postgres

🔐 Generating replicator client certificate for worker-01...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:45:46 UTC] USER=www-data EUID=0 PID=862286 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-02-05 10:45:46 UTC] USER=www-data EUID=0 PID=862296 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-02-05 10:45:46 UTC] USER=www-data EUID=0 PID=862305 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-02-05 10:45:46 UTC] USER=www-data EUID=0 PID=862314 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-02-05 10:45:46 UTC] USER=www-data EUID=0 PID=862323 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:45:46 UTC] USER=www-data EUID=0 PID=862348 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:45:46 UTC] USER=www-data EUID=0 PID=862357 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:45:46 UTC] USER=www-data EUID=0 PID=862366 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:45:46 UTC] USER=www-data EUID=0 PID=862385 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:45:46 UTC] USER=www-data EUID=0 PID=862395 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:45:46 UTC] USER=www-data EUID=0 PID=862417 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862426 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_pk8.der /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862435 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862444 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862453 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862462 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862471 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862480 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862489 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862498 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862507 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862516 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862525 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862534 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862560 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862569 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862578 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862587 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862604 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862636 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862656 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862666 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862684 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862700 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862709 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:45:47 UTC] USER=www-data EUID=0 PID=862737 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862748 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862757 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862767 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862776 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862785 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862794 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862814 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862824 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862842 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862866 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862876 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862885 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862900 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862909 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862918 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862929 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862938 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862950 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:45:48 UTC] USER=www-data EUID=0 PID=862959 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:45:49 UTC] USER=www-data EUID=0 PID=862980 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:45:49 UTC] USER=www-data EUID=0 PID=862998 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_pk8.der
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:49 UTC] USER=www-data EUID=0 PID=863054 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:49 UTC] USER=www-data EUID=0 PID=863094 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:49 UTC] USER=www-data EUID=0 PID=863103 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:45:49 UTC] USER=www-data EUID=0 PID=863113 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:45:49 UTC] USER=www-data EUID=0 PID=863138 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt
[2026-02-05 10:45:49 UTC] USER=www-data EUID=0 PID=863148 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:45:50 UTC] USER=www-data EUID=0 PID=863161 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:45:50 UTC] USER=www-data EUID=0 PID=863175 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 10:45:50 UTC] USER=www-data EUID=0 PID=863185 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:45:50 UTC] USER=www-data EUID=0 PID=863196 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:45:50 UTC] USER=www-data EUID=0 PID=863211 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:45:50 UTC] USER=www-data EUID=0 PID=863236 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:45:51 UTC] USER=www-data EUID=0 PID=863250 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/zone-universe-main-dev
   ✅ Symlinked ca.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 10:45:51 UTC] USER=www-data EUID=0 PID=863301 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 10:45:51 UTC] USER=www-data EUID=0 PID=863314 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:45:51 UTC] USER=www-data EUID=0 PID=863351 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:45:52 UTC] USER=www-data EUID=0 PID=863360 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: zone-universe-main-dev
User: replicator
Node: worker-01
FQDN: db-zone-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-zone-universe-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

✅ Replicator certificate generated for worker-01
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    zone
  Zone:       universe
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/zone/universe/main/dev/postgresql/worker-01/fastorder_admin_gd

Connection Info:
  Socket Dir: /var/run/postgresql-zone-universe-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: fastorder_admin_gd
Storing password in AWS Secrets Manager: fastorder/db/zone/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/zone/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/zone/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Secret created: fastorder/db/zone/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/zone/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/zone/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: worker-01/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Username:    fastorder_admin_gd
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   fastorder_admin_gd
  Hostname:    db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:46:03 UTC] USER=www-data EUID=0 PID=864141 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-fastorder_admin_gd
[2026-02-05 10:46:03 UTC] USER=www-data EUID=0 PID=864152 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-02-05 10:46:03 UTC] USER=www-data EUID=0 PID=864161 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
[2026-02-05 10:46:04 UTC] USER=www-data EUID=0 PID=864174 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-02-05 10:46:04 UTC] USER=www-data EUID=0 PID=864183 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:46:04 UTC] USER=www-data EUID=0 PID=864201 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:46:04 UTC] USER=www-data EUID=0 PID=864210 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:46:04 UTC] USER=www-data EUID=0 PID=864219 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 10:46:04 UTC] USER=www-data EUID=0 PID=864228 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 10:46:04 UTC] USER=www-data EUID=0 PID=864237 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:46:04 UTC] USER=www-data EUID=0 PID=864248 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:46:04 UTC] USER=www-data EUID=0 PID=864261 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 10:46:04 UTC] USER=www-data EUID=0 PID=864275 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 10:46:04 UTC] USER=www-data EUID=0 PID=864291 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_pk8.der /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 10:46:04 UTC] USER=www-data EUID=0 PID=864302 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 10:46:04 UTC] USER=www-data EUID=0 PID=864311 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864320 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864329 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864338 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864347 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864357 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864366 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864375 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864392 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864401 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864410 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864458 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864467 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864476 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864485 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864494 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864503 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864512 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864521 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864530 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864551 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864560 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864570 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864580 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864589 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:46:05 UTC] USER=www-data EUID=0 PID=864600 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864616 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864630 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864641 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864659 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864674 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864692 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864702 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864713 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864722 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864742 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864751 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864760 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864769 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864779 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864788 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864797 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864806 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864816 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864825 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864835 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864844 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864854 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864864 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:46:06 UTC] USER=www-data EUID=0 PID=864874 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=864884 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=864897 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=864909 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=864918 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=864927 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=864936 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=864945 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=864954 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=864966 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=864975 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=864985 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=864995 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=865013 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=865027 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=865047 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=865057 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=865080 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=865118 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:46:07 UTC] USER=www-data EUID=0 PID=865129 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: zone-universe-main-dev
User: fastorder_admin_gd
Node: worker-01
FQDN: db-zone-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-zone-universe-main-dev-postgresql-worker-01.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-zone-universe-main-dev-worker-01:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_zone_universe_main_dev_db already exists, skipping creation
[2026-02-05 10:46:08 UTC] USER=www-data EUID=0 PID=865193 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-zone-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /data/postgresql/17/zone-universe-main-dev/worker-01/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-02-05 10:46:08 UTC] USER=www-data EUID=0 PID=865240 ACTION=passthru ARGS=systemctl reload postgresql@zone-universe-main-dev-worker-01.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

=== Pre-flight Checks ===
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
✓ AWS Secrets Manager accessible

=== Retrieving Credentials from AWS ===
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/zone/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/zone/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/zone/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials loaded for worker-01/fastorder_admin_gd: fastorder_admin_gd@db-zone-universe-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_zone_universe_main_dev_db
✓ Credentials retrieved: fastorder_admin_gd@db-zone-universe-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_zone_universe_main_dev_db
╔════════════════════════════════════════════╗
║  PostgreSQL Test Suite (AWS Secrets MGR)  ║
╚════════════════════════════════════════════╝

=== PostgreSQL Authentication Test ===
✗ PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-zone-universe-main-dev-postgresql-worker-01.fastorder.com" (10.100.1.106), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/zone/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[2026-02-05 10:46:15 UTC] USER=www-data EUID=0 PID=865727 ACTION=fsop ARGS=test -f /data/postgresql/17/zone-universe-main-dev/worker-01/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : zone-universe-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-zone-universe-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_zone_universe_main_dev_db
  SCHEMA      : zone
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
  DNS → 10.100.1.106
  CA         : /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    zone
  Zone:       universe
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/zone/universe/main/dev/postgresql/worker-01/debezium_user

Connection Info:
  Socket Dir: /var/run/postgresql-zone-universe-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User debezium_user does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: debezium_user
Storing password in AWS Secrets Manager: fastorder/db/zone/universe/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/zone/universe/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/zone/universe/main/dev/postgresql/worker-01/debezium_user
✅ Secret created: fastorder/db/zone/universe/main/dev/postgresql/worker-01/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/zone/universe/main/dev/postgresql/worker-01/debezium_user
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/zone/universe/main/dev/postgresql/worker-01/debezium_user
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: worker-01/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-02-05 10:46:26 UTC] USER=www-data EUID=0 PID=866387 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-zone-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_zone_universe_main_dev_db already exists
[2026-02-05 10:46:27 UTC] USER=www-data EUID=0 PID=866429 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-zone-universe-main-dev-worker-01 -p 5432 -d fastorder_zone_universe_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/zone-universe-main-dev/worker-01/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
🔐 Creating replicator role for worker-01...
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : zone-universe-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-zone-universe-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SSL DIR     : /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
  DNS → 10.100.1.106
  CA         : /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
🔑 Generating new secure replicator password...
💾 Storing replicator password in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/db/zone/universe/main/dev/postgresql/replicator-ZxI8Cl",
    "Name": "fastorder/db/zone/universe/main/dev/postgresql/replicator",
    "VersionId": "d593ccb3-4831-41e1-a166-eb7b1f23958d"
}
✅ Password stored in AWS Secrets Manager: fastorder/db/zone/universe/main/dev/postgresql/replicator
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Creating role: replicator with password
SET
CREATE ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/zone/universe/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🎉 Done.
✅ Replicator role created for worker-01

[DEBUG] Tracking substep start: steps/01-install/steps/04-install-extensions (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 04 install extensions...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

╔══════════════════════════════════════════════════════════════╗
║       PostgreSQL Extensions Installation                     ║
╠══════════════════════════════════════════════════════════════╣
║  Environment:  zone-universe-main-dev
║  Identifier:   worker-01
║  PG Version:   17
║  PostGIS:      yes
║  Citus:        yes
╚══════════════════════════════════════════════════════════════╝

[INFO] Installing PostgreSQL extension packages via wrapper...
[INFO] Updating package lists...
[2026-02-05 10:46:35 UTC] USER=www-data EUID=0 PID=866914 ACTION=passthru ARGS=apt-get update -qq
[WARN] apt-get update had warnings (continuing)
[INFO] Installing postgresql-17-pgvector...
[2026-02-05 10:46:35 UTC] USER=www-data EUID=0 PID=866939 ACTION=passthru ARGS=apt-get install -y -qq postgresql-17-pgvector
[WARN] Failed to install postgresql-17-pgvector (may not be available for PG 17)
[OK] Package postgresql-17-postgis-3 already installed
[OK] Package postgresql-17-postgis-3-scripts already installed
[INFO] Database connection:
[INFO]   Host: db-zone-universe-main-dev-postgresql-worker-01.fastorder.com:5432
[INFO]   Database: fastorder_zone_universe_main_dev_db
[INFO]   SSL Dir: /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[INFO] Checking PostgreSQL connectivity...
[OK] PostgreSQL is accessible
[INFO] Installing extensions in postgres database...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Installing Extensions in: postgres
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Installing hstore in postgres...
CREATE EXTENSION
[OK] Installed hstore in postgres
[INFO] Installing vector in postgres...
[WARN] Failed to install vector in postgres (may require superuser or package)
[INFO] Installing ltree in postgres...
CREATE EXTENSION
[OK] Installed ltree in postgres
[INFO] Installing dblink in postgres...
CREATE EXTENSION
[OK] Installed dblink in postgres
[INFO] Installing fuzzystrmatch in postgres...
CREATE EXTENSION
[OK] Installed fuzzystrmatch in postgres
[INFO] Installing pgcrypto in postgres...
CREATE EXTENSION
[OK] Installed pgcrypto in postgres
[INFO] Installing btree_gist in postgres...
CREATE EXTENSION
[OK] Installed btree_gist in postgres
[INFO] Installing pg_trgm in postgres...
CREATE EXTENSION
[OK] Installed pg_trgm in postgres
[INFO] Installing btree_gin in postgres...
CREATE EXTENSION
[OK] Installed btree_gin in postgres
[INFO] Installing uuid-ossp in postgres...
CREATE EXTENSION
[OK] Installed uuid-ossp in postgres
[INFO] Installing unaccent in postgres...
CREATE EXTENSION
[OK] Installed unaccent in postgres
[INFO] Installing PostGIS extensions...
[INFO] Installing postgis in postgres...
CREATE EXTENSION
[OK] Installed postgis in postgres
[INFO] Installing postgis_topology in postgres...
CREATE EXTENSION
[OK] Installed postgis_topology in postgres
[INFO] Installing extensions in fastorder_zone_universe_main_dev_db...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Installing Extensions in: fastorder_zone_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Installing hstore in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed hstore in fastorder_zone_universe_main_dev_db
[INFO] Installing vector in fastorder_zone_universe_main_dev_db...
[WARN] Failed to install vector in fastorder_zone_universe_main_dev_db (may require superuser or package)
[INFO] Installing ltree in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed ltree in fastorder_zone_universe_main_dev_db
[INFO] Installing dblink in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed dblink in fastorder_zone_universe_main_dev_db
[INFO] Installing fuzzystrmatch in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed fuzzystrmatch in fastorder_zone_universe_main_dev_db
[INFO] Installing pgcrypto in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed pgcrypto in fastorder_zone_universe_main_dev_db
[INFO] Installing btree_gist in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed btree_gist in fastorder_zone_universe_main_dev_db
[INFO] Installing pg_trgm in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed pg_trgm in fastorder_zone_universe_main_dev_db
[INFO] Installing btree_gin in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed btree_gin in fastorder_zone_universe_main_dev_db
[INFO] Installing uuid-ossp in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed uuid-ossp in fastorder_zone_universe_main_dev_db
[INFO] Installing unaccent in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed unaccent in fastorder_zone_universe_main_dev_db
[INFO] Installing PostGIS extensions...
[INFO] Installing postgis in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed postgis in fastorder_zone_universe_main_dev_db
[INFO] Installing postgis_topology in fastorder_zone_universe_main_dev_db...
CREATE EXTENSION
[OK] Installed postgis_topology in fastorder_zone_universe_main_dev_db

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Extension Verification: postgres
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ✓ btree_gin        | 1.3
  ✓ btree_gist       | 1.7
  ✓ dblink           | 1.2
  ✓ fuzzystrmatch    | 1.2
  ✓ hstore           | 1.8
  ✓ ltree            | 1.3
  ✓ pg_trgm          | 1.6
  ✓ pgcrypto         | 1.3
  ✓ plpgsql          | 1.0
  ✓ postgis          | 3.5.3
  ✓ postgis_topology | 3.5.3
  ✓ unaccent         | 1.1
  ✓ uuid-ossp        | 1.1


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Extension Verification: fastorder_zone_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ✓ btree_gin        | 1.3
  ✓ btree_gist       | 1.7
  ✓ dblink           | 1.2
  ✓ fuzzystrmatch    | 1.2
  ✓ hstore           | 1.8
  ✓ ltree            | 1.3
  ✓ pg_trgm          | 1.6
  ✓ pgcrypto         | 1.3
  ✓ plpgsql          | 1.0
  ✓ postgis          | 3.5.3
  ✓ postgis_topology | 3.5.3
  ✓ unaccent         | 1.1
  ✓ uuid-ossp        | 1.1


╔══════════════════════════════════════════════════════════════╗
║  ✅ PostgreSQL Extensions Installation Complete              ║
╚══════════════════════════════════════════════════════════════╝


[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (zone) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=zone
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/zone/run.sh
🔍 DEBUG_CHECKPOINT_03: No specific folder for zone, using default
[DEBUG] Tracking substep start: steps/01-install/steps/default (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 🔸 Service: zone (using default contracts schema)
🔍 DEBUG_CHECKPOINT_04: Executing default: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/run.sh
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] 🟢 Starting default contracts schema provisioning for SERVICE=zone
[INFO] Environment: zone-universe-main-dev
[INFO] Schema: zone (contracts tables)
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Skipping Schema Setup on worker-01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

ℹ️  Schema setup only runs on coordinator
ℹ️  This is a worker-01 node - schemas replicate automatically

✅ Nothing to do on this node

✓ ✅ Worker worker-01 setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up standby replicas (1 per worker)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up standby: worker-01-standby-01 (replica of worker-01)
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-02-05 10:46:50 UTC] USER=unknown EUID=33 PID=867783 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 10:46:50 UTC] USER=unknown EUID=33 PID=867790 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 10:46:50 UTC] USER=unknown EUID=33 PID=867802 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-02-05 10:46:50 UTC] USER=unknown EUID=33 PID=867809 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 10:46:50 UTC] USER=unknown EUID=33 PID=867816 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-02-05 10:46:50 UTC] USER=unknown EUID=33 PID=867823 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for zone in universe-dev...
[INFO] Environment: zone-universe-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172 JOB_UUID=20cfe3cc-b016-4175-bac8-362f4003c72d

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[WARN] Could not find PostgreSQL IP for worker-01-standby-01 in topology.json, allocating new VM IP...
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Allocated new VM IP: 10.100.1.196 for db-worker-01-standby-01-postgresql
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: zone-universe-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] PostgreSQL IP: 10.100.1.196
[INFO] Primary hostname: db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01-standby-01...
[INFO]   db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.196

[INFO]   ➕ Adding db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.196
✅     ✅ Added: db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.196

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.196    db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  zone-universe-main-dev.fastorder.com
  Alt CN:      zone-universe-main-dev.fastorder.com
  Internal IP: 10.100.1.196 (from topology: db-worker-01-standby-01-postgresql)
  Shared IP:   142.93.238.16 (legacy/public, also included in SANs)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 10:46:55 UTC] USER=www-data EUID=0 PID=868407 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:46:55 UTC] USER=www-data EUID=0 PID=868416 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-02-05 10:46:55 UTC] USER=www-data EUID=0 PID=868429 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-868366
[2026-02-05 10:46:55 UTC] USER=www-data EUID=0 PID=868438 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-868366/ra_root.crt
[2026-02-05 10:46:55 UTC] USER=www-data EUID=0 PID=868450 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-868366/ra_root.key
[2026-02-05 10:46:55 UTC] USER=www-data EUID=0 PID=868461 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-868366/ra_root.crt
[2026-02-05 10:46:55 UTC] USER=www-data EUID=0 PID=868472 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-868366/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = zone-universe-main-dev.fastorder.com
[2026-02-05 10:46:58 UTC] USER=www-data EUID=0 PID=868583 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-868366/server.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.key
[2026-02-05 10:46:58 UTC] USER=www-data EUID=0 PID=868593 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-868366/server.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt
[2026-02-05 10:46:58 UTC] USER=www-data EUID=0 PID=868609 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-02-05 10:46:58 UTC] USER=www-data EUID=0 PID=868624 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-868366/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:46:58 UTC] USER=www-data EUID=0 PID=868634 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:46:58 UTC] USER=www-data EUID=0 PID=868654 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:46:58 UTC] USER=www-data EUID=0 PID=868671 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 10:46:58 UTC] USER=www-data EUID=0 PID=868691 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.key
[2026-02-05 10:46:58 UTC] USER=www-data EUID=0 PID=868701 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.key
[2026-02-05 10:46:58 UTC] USER=www-data EUID=0 PID=868713 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt
[2026-02-05 10:46:58 UTC] USER=www-data EUID=0 PID=868722 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt
[2026-02-05 10:46:58 UTC] USER=www-data EUID=0 PID=868731 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:46:58 UTC] USER=www-data EUID=0 PID=868740 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = zone-universe-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:zone-universe-main-dev.fastorder.com, DNS:zone-universe-main-dev.fastorder.com, DNS:db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-zone-universe-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:10.100.1.196, IP Address:127.0.0.1, IP Address:142.93.238.16
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: zone-universe-main-dev
Node:        worker-01-standby-01
Primary CN:  zone-universe-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@zone-universe-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=zone-universe-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:46:59 UTC] USER=www-data EUID=0 PID=868828 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-02-05 10:46:59 UTC] USER=www-data EUID=0 PID=868837 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-02-05 10:46:59 UTC] USER=www-data EUID=0 PID=868846 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
[2026-02-05 10:46:59 UTC] USER=www-data EUID=0 PID=868858 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-02-05 10:46:59 UTC] USER=www-data EUID=0 PID=868870 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:46:59 UTC] USER=www-data EUID=0 PID=868895 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:46:59 UTC] USER=www-data EUID=0 PID=868904 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:46:59 UTC] USER=www-data EUID=0 PID=868913 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key
[2026-02-05 10:46:59 UTC] USER=www-data EUID=0 PID=868922 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-02-05 10:46:59 UTC] USER=www-data EUID=0 PID=868931 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:46:59 UTC] USER=www-data EUID=0 PID=868945 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/ca.crt
[2026-02-05 10:46:59 UTC] USER=www-data EUID=0 PID=868963 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-02-05 10:46:59 UTC] USER=www-data EUID=0 PID=868979 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-02-05 10:46:59 UTC] USER=www-data EUID=0 PID=868992 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-02-05 10:46:59 UTC] USER=www-data EUID=0 PID=869007 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key
[2026-02-05 10:47:00 UTC] USER=www-data EUID=0 PID=869025 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:47:00 UTC] USER=www-data EUID=0 PID=869034 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:00 UTC] USER=www-data EUID=0 PID=869047 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key
[2026-02-05 10:47:00 UTC] USER=www-data EUID=0 PID=869059 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-02-05 10:47:00 UTC] USER=www-data EUID=0 PID=869068 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-02-05 10:47:00 UTC] USER=www-data EUID=0 PID=869077 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:00 UTC] USER=www-data EUID=0 PID=869138 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:00 UTC] USER=www-data EUID=0 PID=869147 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:00 UTC] USER=www-data EUID=0 PID=869156 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:00 UTC] USER=www-data EUID=0 PID=869177 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:00 UTC] USER=www-data EUID=0 PID=869202 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-02-05 10:47:01 UTC] USER=www-data EUID=0 PID=869212 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:47:01 UTC] USER=www-data EUID=0 PID=869222 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/ca.crt
[2026-02-05 10:47:01 UTC] USER=www-data EUID=0 PID=869231 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-02-05 10:47:01 UTC] USER=www-data EUID=0 PID=869241 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-02-05 10:47:01 UTC] USER=www-data EUID=0 PID=869261 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:01 UTC] USER=www-data EUID=0 PID=869271 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:01 UTC] USER=www-data EUID=0 PID=869280 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:01 UTC] USER=www-data EUID=0 PID=869291 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:01 UTC] USER=www-data EUID=0 PID=869300 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:01 UTC] USER=www-data EUID=0 PID=869313 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:01 UTC] USER=www-data EUID=0 PID=869363 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-02-05 10:47:02 UTC] USER=www-data EUID=0 PID=869375 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:47:02 UTC] USER=www-data EUID=0 PID=869385 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/ca.crt
[2026-02-05 10:47:02 UTC] USER=www-data EUID=0 PID=869416 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-02-05 10:47:02 UTC] USER=www-data EUID=0 PID=869443 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-02-05 10:47:02 UTC] USER=www-data EUID=0 PID=869465 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:02 UTC] USER=www-data EUID=0 PID=869476 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:02 UTC] USER=www-data EUID=0 PID=869485 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:02 UTC] USER=www-data EUID=0 PID=869552 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869616 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869626 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869635 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869644 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869654 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/ca.crt
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869666 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869681 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869692 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869703 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869713 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869722 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869731 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869741 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869751 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869763 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869773 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869785 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869798 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/ca.crt
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869808 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869817 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_der.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-02-05 10:47:03 UTC] USER=www-data EUID=0 PID=869826 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-02-05 10:47:04 UTC] USER=www-data EUID=0 PID=869836 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:47:04 UTC] USER=www-data EUID=0 PID=869849 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:47:04 UTC] USER=www-data EUID=0 PID=869858 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:47:04 UTC] USER=www-data EUID=0 PID=869867 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 10:47:04 UTC] USER=www-data EUID=0 PID=869876 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 10:47:04 UTC] USER=www-data EUID=0 PID=869885 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 10:47:04 UTC] USER=www-data EUID=0 PID=869894 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:47:04 UTC] USER=www-data EUID=0 PID=869903 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:47:04 UTC] USER=www-data EUID=0 PID=869912 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:47:04 UTC] USER=www-data EUID=0 PID=869921 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: zone-universe-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:47:04 UTC] USER=www-data EUID=0 PID=869978 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-02-05 10:47:04 UTC] USER=www-data EUID=0 PID=869987 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-02-05 10:47:04 UTC] USER=www-data EUID=0 PID=869997 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
[2026-02-05 10:47:04 UTC] USER=www-data EUID=0 PID=870006 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-02-05 10:47:04 UTC] USER=www-data EUID=0 PID=870016 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870039 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870055 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870066 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870077 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870086 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870095 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/ca.crt
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870104 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870113 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870122 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870131 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870140 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870149 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870158 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870167 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870176 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870185 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870194 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870203 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870212 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870221 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870230 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870256 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:05 UTC] USER=www-data EUID=0 PID=870265 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870274 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870285 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870296 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870305 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870314 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870324 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870337 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/ca.crt
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870348 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870377 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870392 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870402 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870425 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870435 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870451 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870462 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870472 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870481 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870490 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870499 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870508 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/ca.crt
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870517 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870526 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870536 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870548 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870576 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870585 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:06 UTC] USER=www-data EUID=0 PID=870594 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870603 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870619 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870628 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870639 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870648 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870657 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/ca.crt
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870666 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870675 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870684 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870694 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870708 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870717 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870726 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870735 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870745 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870754 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870763 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870772 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870781 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/ca.crt
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870790 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870799 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_der.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870808 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870818 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870828 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870837 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870846 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870855 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870864 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870877 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870890 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870899 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:47:07 UTC] USER=www-data EUID=0 PID=870908 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: zone-universe-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:47:08 UTC] USER=www-data EUID=0 PID=870954 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-02-05 10:47:08 UTC] USER=www-data EUID=0 PID=870964 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-02-05 10:47:08 UTC] USER=www-data EUID=0 PID=870987 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-02-05 10:47:08 UTC] USER=www-data EUID=0 PID=870996 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:47:08 UTC] USER=www-data EUID=0 PID=871012 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:47:08 UTC] USER=www-data EUID=0 PID=871021 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:47:08 UTC] USER=www-data EUID=0 PID=871030 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:47:08 UTC] USER=www-data EUID=0 PID=871039 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt
[2026-02-05 10:47:08 UTC] USER=www-data EUID=0 PID=871048 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871057 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871067 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871078 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871104 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_pk8.der /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871113 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871131 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871140 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871149 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871158 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871167 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871176 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871185 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871194 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871203 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871212 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871253 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871269 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871284 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871294 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871313 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871324 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871334 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871343 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871352 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871361 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871370 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871379 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:47:09 UTC] USER=www-data EUID=0 PID=871389 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871411 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871422 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871431 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871440 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871450 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871459 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871468 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871491 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871500 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871509 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871519 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871530 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871540 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871549 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871559 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871570 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871579 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871588 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871597 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871606 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:47:10 UTC] USER=www-data EUID=0 PID=871619 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871630 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871639 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871649 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871659 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871668 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871677 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871689 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871698 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871716 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871734 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871743 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871754 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871765 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871775 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871785 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871803 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871824 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871841 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871850 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:47:11 UTC] USER=www-data EUID=0 PID=871876 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: zone-universe-main-dev
User: replicator
Node: worker-01
FQDN: db-zone-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-zone-universe-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-standby-01-postgresql environment: db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com (10.100.1.196)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.196
[INFO] Environment: zone-universe-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] Data dir:   /data/postgresql/17/zone-universe-main-dev/worker-01-standby-01
[INFO] Port:       5432
[INFO] Hostname:   db-zone-universe-main-dev-postgresql-worker-01-standby-01
[2026-02-05 10:47:13 UTC] USER=www-data EUID=0 PID=872117 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:14 UTC] USER=www-data EUID=0 PID=872140 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:14 UTC] USER=www-data EUID=0 PID=872165 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  zone-universe-main-dev.fastorder.com
  Alt CN:      zone-universe-main-dev.fastorder.com
  Internal IP: 10.100.1.196 (from topology: db-worker-01-standby-01-postgresql)
  Shared IP:   142.93.238.16 (legacy/public, also included in SANs)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 10:47:14 UTC] USER=www-data EUID=0 PID=872235 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:14 UTC] USER=www-data EUID=0 PID=872259 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-02-05 10:47:14 UTC] USER=www-data EUID=0 PID=872271 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-872172
[2026-02-05 10:47:14 UTC] USER=www-data EUID=0 PID=872281 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-872172/ra_root.crt
[2026-02-05 10:47:14 UTC] USER=www-data EUID=0 PID=872290 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-872172/ra_root.key
[2026-02-05 10:47:14 UTC] USER=www-data EUID=0 PID=872299 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-872172/ra_root.crt
[2026-02-05 10:47:14 UTC] USER=www-data EUID=0 PID=872309 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-872172/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = zone-universe-main-dev.fastorder.com
[2026-02-05 10:47:17 UTC] USER=www-data EUID=0 PID=872485 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-872172/server.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.key
[2026-02-05 10:47:17 UTC] USER=www-data EUID=0 PID=872498 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-872172/server.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt
[2026-02-05 10:47:17 UTC] USER=www-data EUID=0 PID=872516 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-02-05 10:47:17 UTC] USER=www-data EUID=0 PID=872528 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-872172/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:47:17 UTC] USER=www-data EUID=0 PID=872542 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:47:17 UTC] USER=www-data EUID=0 PID=872551 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt
[2026-02-05 10:47:18 UTC] USER=www-data EUID=0 PID=872564 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 10:47:18 UTC] USER=www-data EUID=0 PID=872577 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.key
[2026-02-05 10:47:18 UTC] USER=www-data EUID=0 PID=872586 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.key
[2026-02-05 10:47:18 UTC] USER=www-data EUID=0 PID=872598 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt
[2026-02-05 10:47:18 UTC] USER=www-data EUID=0 PID=872617 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
[2026-02-05 10:47:18 UTC] USER=www-data EUID=0 PID=872626 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = zone-universe-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:zone-universe-main-dev.fastorder.com, DNS:zone-universe-main-dev.fastorder.com, DNS:db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-zone-universe-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:10.100.1.196, IP Address:127.0.0.1, IP Address:142.93.238.16
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: zone-universe-main-dev
Node:        worker-01-standby-01
Primary CN:  zone-universe-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@zone-universe-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=zone-universe-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-02-05 10:47:18 UTC] USER=www-data EUID=0 PID=872655 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt
[2026-02-05 10:47:18 UTC] USER=www-data EUID=0 PID=872673 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-02-05 10:47:18 UTC] USER=www-data EUID=0 PID=872694 ACTION=passthru ARGS=systemctl stop postgresql@zone-universe-main-dev-worker-01-standby-01.service
[2026-02-05 10:47:19 UTC] USER=www-data EUID=0 PID=872715 ACTION=passthru ARGS=systemctl stop postgresql
[OK]   No conflicting Postgres left on port 5432
[OK]   Generated new postgres password for initdb
[2026-02-05 10:47:46 UTC] USER=www-data EUID=0 PID=873872 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.q3CFJO
[2026-02-05 10:47:46 UTC] USER=www-data EUID=0 PID=873893 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.q3CFJO
[2026-02-05 10:47:46 UTC] USER=www-data EUID=0 PID=873915 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/zone-universe-main-dev
[2026-02-05 10:47:46 UTC] USER=www-data EUID=0 PID=873938 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/zone-universe-main-dev
[2026-02-05 10:47:47 UTC] USER=www-data EUID=0 PID=873960 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/zone-universe-main-dev
[INFO] This is a standby. Using pg_basebackup from primary (worker-01)...
[INFO] Setting up replicator role and slot on primary (worker-01)...
ℹ️  Scanning primary for stuck queries from previous failed attempts...
ℹ️  Scanning for stuck queries (timeout: 30s)...
ℹ️  No stuck queries found
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : zone-universe-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-zone-universe-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SLOT        : worker_01_standby_01
  SSL DIR     : /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
  DNS → 10.100.1.106
  CA         : /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Role replicator already exists, updating password and ensuring REPLICATION privilege
SET
ALTER ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/zone/universe/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🔧 Ensuring replication slot: worker_01_standby_01…
🆕 Creating replication slot worker_01_standby_01
SET
 pg_create_physical_replication_slot 
-------------------------------------
 (worker_01_standby_01,)
(1 row)

✅ Replication slot worker_01_standby_01 created.
🎉 Done.
[OK]   Replicator role and slot created on primary
[INFO] Creating replicator client certificates for connecting to primary (worker-01)...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:47:50 UTC] USER=www-data EUID=0 PID=874194 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-02-05 10:47:50 UTC] USER=www-data EUID=0 PID=874203 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-02-05 10:47:50 UTC] USER=www-data EUID=0 PID=874212 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-02-05 10:47:50 UTC] USER=www-data EUID=0 PID=874221 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-02-05 10:47:50 UTC] USER=www-data EUID=0 PID=874230 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:47:50 UTC] USER=www-data EUID=0 PID=874251 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:47:50 UTC] USER=www-data EUID=0 PID=874260 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:47:50 UTC] USER=www-data EUID=0 PID=874269 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:47:50 UTC] USER=www-data EUID=0 PID=874278 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt
[2026-02-05 10:47:51 UTC] USER=www-data EUID=0 PID=874287 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:47:51 UTC] USER=www-data EUID=0 PID=874296 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:47:51 UTC] USER=www-data EUID=0 PID=874310 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 10:47:51 UTC] USER=www-data EUID=0 PID=874319 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:47:51 UTC] USER=www-data EUID=0 PID=874328 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_pk8.der /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:47:51 UTC] USER=www-data EUID=0 PID=874341 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:47:51 UTC] USER=www-data EUID=0 PID=874368 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:47:51 UTC] USER=www-data EUID=0 PID=874377 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:47:51 UTC] USER=www-data EUID=0 PID=874386 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[2026-02-05 10:47:51 UTC] USER=www-data EUID=0 PID=874395 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:47:51 UTC] USER=www-data EUID=0 PID=874404 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 10:47:51 UTC] USER=www-data EUID=0 PID=874413 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:47:51 UTC] USER=www-data EUID=0 PID=874422 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:47:51 UTC] USER=www-data EUID=0 PID=874431 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:47:51 UTC] USER=www-data EUID=0 PID=874440 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:51 UTC] USER=www-data EUID=0 PID=874488 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:51 UTC] USER=www-data EUID=0 PID=874512 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874532 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874547 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874565 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874574 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874584 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874593 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874603 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874612 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874622 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874633 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874643 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874652 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874661 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874671 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874681 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874693 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874704 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874717 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874726 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874735 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874745 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874774 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874784 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:52 UTC] USER=www-data EUID=0 PID=874793 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:53 UTC] USER=www-data EUID=0 PID=874810 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:53 UTC] USER=www-data EUID=0 PID=874839 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:53 UTC] USER=www-data EUID=0 PID=874859 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:53 UTC] USER=www-data EUID=0 PID=874872 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:47:53 UTC] USER=www-data EUID=0 PID=874888 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt
[2026-02-05 10:47:53 UTC] USER=www-data EUID=0 PID=874925 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:47:53 UTC] USER=www-data EUID=0 PID=874936 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 10:47:53 UTC] USER=www-data EUID=0 PID=874963 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:47:53 UTC] USER=www-data EUID=0 PID=874974 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:53 UTC] USER=www-data EUID=0 PID=874984 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:54 UTC] USER=www-data EUID=0 PID=874997 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:54 UTC] USER=www-data EUID=0 PID=875006 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:54 UTC] USER=www-data EUID=0 PID=875025 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:54 UTC] USER=www-data EUID=0 PID=875034 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:47:54 UTC] USER=www-data EUID=0 PID=875052 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:47:54 UTC] USER=www-data EUID=0 PID=875061 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
[2026-02-05 10:47:54 UTC] USER=www-data EUID=0 PID=875070 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 10:47:54 UTC] USER=www-data EUID=0 PID=875079 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 10:47:54 UTC] USER=www-data EUID=0 PID=875088 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator_pk8.der /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 10:47:54 UTC] USER=www-data EUID=0 PID=875098 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:47:54 UTC] USER=www-data EUID=0 PID=875110 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:47:54 UTC] USER=www-data EUID=0 PID=875126 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:47:55 UTC] USER=www-data EUID=0 PID=875136 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 10:47:55 UTC] USER=www-data EUID=0 PID=875152 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
   ✅ Symlinked client-key.pem
[2026-02-05 10:47:55 UTC] USER=www-data EUID=0 PID=875173 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:47:55 UTC] USER=www-data EUID=0 PID=875184 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:47:55 UTC] USER=www-data EUID=0 PID=875193 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:47:55 UTC] USER=www-data EUID=0 PID=875202 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: zone-universe-main-dev
User: replicator
Node: worker-01
FQDN: db-zone-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-zone-universe-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

[OK]   Replicator certificate created for worker-01 in /home/postgres/
[INFO] Using replicator certificates from primary worker-01...
[2026-02-05 10:47:55 UTC] USER=www-data EUID=0 PID=875235 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 10:47:56 UTC] USER=www-data EUID=0 PID=875275 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.key
[2026-02-05 10:47:56 UTC] USER=www-data EUID=0 PID=875296 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/replicator.crt
[OK]   Replicator certificates verified at /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[OK]   root.crt verified at /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01
[INFO] Updating primary pg_hba.conf to allow replication...
[INFO]   Standby IP: 10.100.1.196/32 (standby's source IP)
[INFO]   Primary application IP: 10.100.1.106/32 (for local pg_basebackup)
[INFO]   Primary DNS IP: 10.100.1.106/32 (DNS resolution of db-zone-universe-main-dev-postgresql-worker-01.fastorder.com)
[2026-02-05 10:47:56 UTC] USER=www-data EUID=0 PID=875383 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.196/32  scram-sha-256 
      $0==begin {inside=1}
      inside && $0==rule {found=1}
      $0==end {inside=0}
      END {exit found?0:1}
     /data/postgresql/17/zone-universe-main-dev/worker-01/pg_hba.conf
[2026-02-05 10:47:56 UTC] USER=www-data EUID=0 PID=875415 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl  replication  replicator  10.100.1.196/32  scram-sha-256 /data/postgresql/17/zone-universe-main-dev/worker-01/pg_hba.conf
[2026-02-05 10:47:56 UTC] USER=www-data EUID=0 PID=875437 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.106/32  scram-sha-256 
        $0==begin {inside=1}
        inside && $0==rule {found=1}
        $0==end {inside=0}
        END {exit found?0:1}
       /data/postgresql/17/zone-universe-main-dev/worker-01/pg_hba.conf
[2026-02-05 10:47:57 UTC] USER=www-data EUID=0 PID=875464 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl  replication  replicator  10.100.1.106/32  scram-sha-256 /data/postgresql/17/zone-universe-main-dev/worker-01/pg_hba.conf
[INFO] Reloading primary PostgreSQL service...
[2026-02-05 10:47:57 UTC] USER=www-data EUID=0 PID=875485 ACTION=passthru ARGS=systemctl reload postgresql@zone-universe-main-dev-worker-01.service
[OK]   Primary pg_hba.conf updated and service reloaded
[INFO] Primary host: db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
[INFO] Using replicator cert: /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt
[INFO] Using replicator key: /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key (PKCS#8 format)
[INFO] Using CA cert: /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[INFO] Verifying postgres user can access certificates...
[ERR]  postgres user CANNOT read /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[INFO] File permissions:
lrwxrwxrwx 1 postgres ssl-cert 73 Feb  5 10:47 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt -> /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/root.crt
[INFO] Parent directory permissions:
drwx------ 2 postgres postgres 4096 Feb  5 10:47 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
drwx------ 5 postgres postgres 4096 Feb  5 10:47 /home/postgres/ssl/.postgresql/zone-universe-main-dev
[WARN] Attempting to fix permissions (/usr/local/bin/fastorder-provisioning-wrapper.sh required)...
[INFO] Fixing /home/postgres/ directory...
[2026-02-05 10:47:58 UTC] USER=www-data EUID=0 PID=875642 ACTION=fsop ARGS=chmod 755 /home/postgres/
[INFO] Fixing /home/postgres/ssl/.postgresql/...
[2026-02-05 10:47:58 UTC] USER=www-data EUID=0 PID=875668 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/
[INFO] Fixing parent directory: /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:47:58 UTC] USER=www-data EUID=0 PID=875691 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/zone-universe-main-dev
[INFO] Fixing certificate directory: /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[2026-02-05 10:47:58 UTC] USER=www-data EUID=0 PID=875712 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[INFO] Fixing CA certificate: /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[2026-02-05 10:47:58 UTC] USER=www-data EUID=0 PID=875740 ACTION=fsop ARGS=chmod 644 /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[OK]   Permissions fixed
[OK]   postgres user can now read /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt after permission fix
[2026-02-05 10:47:59 UTC] USER=www-data EUID=0 PID=875765 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-zone-universe-main-dev-worker-01-standby-01
[2026-02-05 10:47:59 UTC] USER=www-data EUID=0 PID=875792 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-zone-universe-main-dev-worker-01-standby-01
[2026-02-05 10:47:59 UTC] USER=www-data EUID=0 PID=875814 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-zone-universe-main-dev-worker-01-standby-01
[INFO] Checking primary database size before pg_basebackup...
[INFO] Total primary database size: 47 MB
[INFO] Estimated transfer time: ~0 minutes (at 10MB/s with compression)
[INFO] Retrieving replicator password from AWS Secrets Manager: fastorder/db/zone/universe/main/dev/postgresql/replicator
[OK]   Replicator password retrieved successfully
[INFO] Starting pg_basebackup...
[2026-02-05 10:48:01 UTC] USER=www-data EUID=0 PID=875976 ACTION=passthru ARGS=sudo -u postgres env PGPASSWORD=RCaocde91QfvKauK7zaQwuHCFU4JyAoY PGSSLMODE=verify-full PGSSLCERT=/home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt PGSSLKEY=/home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key PGSSLROOTCERT=/home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt /usr/lib/postgresql/17/bin/pg_basebackup -h db-zone-universe-main-dev-postgresql-worker-01.fastorder.com -p 5432 -U replicator -D /data/postgresql/17/zone-universe-main-dev/worker-01-standby-01 -Fp -Xs -P -R --checkpoint=fast --wal-method=stream --verbose
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/4000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_875987"
34350/49188 kB (69%), 0/1 tablespace (...er-01-standby-01/base/1/2609_fsm)
49199/49199 kB (100%), 0/1 tablespace (...-01-standby-01/global/pg_control)
49199/49199 kB (100%), 1/1 tablespace                                         
pg_basebackup: write-ahead log end point: 0/4000120
pg_basebackup: waiting for background process to finish streaming ...
pg_basebackup: syncing data to disk ...
pg_basebackup: renaming backup_manifest.tmp to backup_manifest
pg_basebackup: base backup completed
[OK]   pg_basebackup complete
[INFO] Fixing postgresql.auto.conf to use IP-based primary_conninfo (matching golden backup)...
[2026-02-05 10:48:03 UTC] USER=www-data EUID=0 PID=876150 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/zone-universe-main-dev/worker-01-standby-01/standby.signal
[2026-02-05 10:48:03 UTC] USER=www-data EUID=0 PID=876178 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/zone-universe-main-dev/worker-01-standby-01/standby.signal
[2026-02-05 10:48:03 UTC] USER=www-data EUID=0 PID=876207 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/zone-universe-main-dev/worker-01-standby-01/standby.signal
[2026-02-05 10:48:03 UTC] USER=www-data EUID=0 PID=876216 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/zone-universe-main-dev/worker-01-standby-01/standby.signal
[OK]   standby.signal verified and permissions set
[INFO] Fixing postgresql.conf with standby-specific settings...
[WARN] postgresql.conf not found at /data/postgresql/17/zone-universe-main-dev/worker-01-standby-01/postgresql.conf
[INFO] Verifying postgresql.auto.conf...
[WARN] postgresql.auto.conf not found - pg_basebackup may have failed
[2026-02-05 10:48:04 UTC] USER=www-data EUID=0 PID=876243 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.q3CFJO
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-02-05 10:48:04 UTC] USER=www-data EUID=0 PID=876294 ACTION=fsop ARGS=cp /tmp/tmp.N9kKzXRyGN /data/postgresql/17/zone-universe-main-dev/worker-01-standby-01/pg_hba.conf
[2026-02-05 10:48:04 UTC] USER=www-data EUID=0 PID=876315 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/zone-universe-main-dev/worker-01-standby-01/pg_hba.conf
[2026-02-05 10:48:04 UTC] USER=www-data EUID=0 PID=876336 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/zone-universe-main-dev/worker-01-standby-01/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@zone-universe-main-dev-worker-01-standby-01.service
[2026-02-05 10:48:04 UTC] USER=www-data EUID=0 PID=876361 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.FgkHRB /etc/systemd/system/postgresql@zone-universe-main-dev-worker-01-standby-01.service
[2026-02-05 10:48:04 UTC] USER=www-data EUID=0 PID=876383 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@zone-universe-main-dev-worker-01-standby-01.service
[OK]   systemd unit written
[2026-02-05 10:48:04 UTC] USER=www-data EUID=0 PID=876404 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 10:48:04 UTC] USER=www-data EUID=0 PID=876425 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 10:48:04 UTC] USER=www-data EUID=0 PID=876447 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-02-05 10:48:06 UTC] USER=www-data EUID=0 PID=876585 ACTION=passthru ARGS=systemctl start postgresql@zone-universe-main-dev-worker-01-standby-01.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-02-05 10:48:07 UTC] USER=www-data EUID=0 PID=876634 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@zone-universe-main-dev-worker-01-standby-01.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Configuring synchronous replication on primary worker-01...
[INFO] Current synchronous_standby_names: ''
[INFO] Initializing synchronous_standby_names with first standby
[INFO] New synchronous_standby_names: 'ANY 1 (worker_01_standby_01)'
[2026-02-05 10:48:07 UTC] USER=www-data EUID=0 PID=876716 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_commit = on;
ALTER SYSTEM
[2026-02-05 10:48:07 UTC] USER=www-data EUID=0 PID=876745 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_standby_names = 'ANY 1 (worker_01_standby_01)';
ALTER SYSTEM
[2026-02-05 10:48:08 UTC] USER=www-data EUID=0 PID=876772 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-zone-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   ✅ Synchronous replication configured on primary
[OK]      Setting: ANY 1 (worker_01_standby_01)
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Skipping database/role provisioning on standby node (read-only)
[INFO]   Database/roles will be replicated from primary: worker-01
[INFO] Applying connection and memory optimizations...
[INFO] Standby will use primary's max_connections: 100
[INFO] Current settings: max_connections=100, work_mem=8MB
[INFO] Target settings (standby): max_connections=100, work_mem=8MB
[OK]   Connection settings already optimized
[INFO] Skipping password setting - this is a standby (read-only)
[INFO] Use primary's postgres password to connect to this standby
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: zone-universe-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] PostgreSQL IP: 10.100.1.196
[INFO] Primary hostname: db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01-standby-01...
[INFO]   db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.196

[INFO]   ✅ db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.196    db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com


[OK]   PostgreSQL 'zone-universe-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.key \
        host=db-zone-universe-main-dev-postgresql-worker-01-standby-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        zone-universe-main-dev-postgresql-worker-01-standby-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.196
[INFO]   Port:              5432
[INFO]   FQDN:              db-zone-universe-main-dev-postgresql-worker-01-standby-01
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 9141c678-a9ea-4c62-992a-6c1b13f6271c
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 03 role...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[2026-02-05 10:48:16 UTC] USER=www-data EUID=0 PID=877498 ACTION=fsop ARGS=test -f /data/postgresql/17/zone-universe-main-dev/worker-01-standby-01/standby.signal
⚠ This is a PostgreSQL STANDBY (read-only replica)
⚠ Skipping role creation - standby gets roles from primary via replication
⚠ Use the PRIMARY's credentials to connect to this standby


[DEBUG] Tracking substep start: steps/01-install/steps/04-install-extensions (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 04 install extensions...
⚠ Standby node - extensions replicate from primary

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (zone) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=zone
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/zone/run.sh
🔍 DEBUG_CHECKPOINT_03: No specific folder for zone, using default
[DEBUG] Tracking substep start: steps/01-install/steps/default (RUN_UUID=118f1726-a0ea-4878-808a-f618237d7172)
[INFO] 🔸 Service: zone (using default contracts schema)
🔍 DEBUG_CHECKPOINT_04: Executing default: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/run.sh
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] 🟢 Starting default contracts schema provisioning for SERVICE=zone
[INFO] Environment: zone-universe-main-dev
[INFO] Schema: zone (contracts tables)
[INFO] Identifier: worker-01-standby-01
[INFO] VM IP: 142.93.238.16

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Skipping Schema Setup on worker-01-standby-01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

ℹ️  Schema setup only runs on coordinator
ℹ️  This is a worker-01-standby-01 node - schemas replicate automatically

✅ Nothing to do on this node

✓ ✅ Standby worker-01-standby-01 setup completed

✓ ✅ PostgreSQL installation completed
[INFO] Discovering additional setup steps...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 02-pg-bouncer.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up PgBouncer connection pooling...
[2026-02-05 10:48:23 UTC] USER=www-data EUID=0 PID=877866 ACTION=fsop ARGS=rm -f /tmp/pgbouncer-ip.service /tmp/pgbouncer.service
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] Checking for existing PgBouncer application environment in topology …
[OK]   Using existing PgBouncer environment:
[INFO]   IP:     10.100.1.78
[INFO]   FQDN:   db-zone-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Domain: db-zone-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO] Ensuring /etc/hosts entry for db-zone-universe-main-dev-postgresql-bouncer.fastorder.com …
[INFO] Adding db-zone-universe-main-dev-postgresql-bouncer.fastorder.com to /etc/hosts
[2026-02-05 10:48:24 UTC] USER=www-data EUID=0 PID=877957 ACTION=fsop ARGS=sed -i /\sdb-zone-universe-main-dev-postgresql-bouncer.fastorder.com\(\s\|$\)/d /etc/hosts
[OK]   Added db-zone-universe-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.78 to /etc/hosts
[INFO] Final verification of /etc/hosts entry for db-zone-universe-main-dev-postgresql-bouncer.fastorder.com …
[OK]   /etc/hosts correctly maps db-zone-universe-main-dev-postgresql-bouncer.fastorder.com to 10.100.1.78
[WARN] IP 10.100.1.78 is already bound to other interface(s):
        inet 10.100.1.78/32 scope global eth0:78
[INFO] Attempting to also bind 10.100.1.78 to lo:pgbouncer ...
[2026-02-05 10:48:24 UTC] USER=www-data EUID=0 PID=877993 ACTION=passthru ARGS=ip addr add 10.100.1.78/32 dev lo label lo:pgbouncer
[OK]   Successfully bound 10.100.1.78 to lo:pgbouncer
[2026-02-05 10:48:25 UTC] USER=www-data EUID=0 PID=878011 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 10:48:26 UTC] USER=www-data EUID=0 PID=878117 ACTION=passthru ARGS=systemctl restart pgbouncer-ip@zone-universe-main-dev.service
Job for pgbouncer-ip@zone-universe-main-dev.service failed because the control process exited with error code.
See "systemctl status pgbouncer-ip@zone-universe-main-dev.service" and "journalctl -xeu pgbouncer-ip@zone-universe-main-dev.service" for details.
[2026-02-05 10:48:26 UTC] USER=www-data EUID=0 PID=878127 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer-ip@zone-universe-main-dev.service
[WARN] pgbouncer-ip@zone-universe-main-dev.service is not active
[WARN] Check status: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@zone-universe-main-dev.service
[2026-02-05 10:48:26 UTC] USER=www-data EUID=0 PID=878151 ACTION=fsop ARGS=mkdir -p /etc/pgbouncer/zone-universe-main-dev
[2026-02-05 10:48:26 UTC] USER=www-data EUID=0 PID=878164 ACTION=fsop ARGS=mkdir -p /run/pgbouncer/zone-universe-main-dev
[2026-02-05 10:48:26 UTC] USER=www-data EUID=0 PID=878173 ACTION=fsop ARGS=mkdir -p /var/log/pgbouncer/zone-universe-main-dev
[2026-02-05 10:48:26 UTC] USER=www-data EUID=0 PID=878182 ACTION=fsop ARGS=chmod 750 /etc/pgbouncer/zone-universe-main-dev
[2026-02-05 10:48:26 UTC] USER=www-data EUID=0 PID=878193 ACTION=fsop ARGS=chmod 750 /run/pgbouncer/zone-universe-main-dev
[2026-02-05 10:48:26 UTC] USER=www-data EUID=0 PID=878202 ACTION=fsop ARGS=chmod 750 /var/log/pgbouncer/zone-universe-main-dev
[2026-02-05 10:48:26 UTC] USER=www-data EUID=0 PID=878211 ACTION=fsop ARGS=chown root:postgres /etc/pgbouncer/zone-universe-main-dev
[2026-02-05 10:48:26 UTC] USER=www-data EUID=0 PID=878221 ACTION=fsop ARGS=chown postgres:postgres /run/pgbouncer/zone-universe-main-dev
[2026-02-05 10:48:26 UTC] USER=www-data EUID=0 PID=878230 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbouncer/zone-universe-main-dev
[INFO] Generating PgBouncer TLS certificate via existing server.sh (IDENTIFIER=pgbouncer) …
[INFO] Calling: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/ssl/server.sh pgbouncer
[INFO] With: OVERRIDE_CN=db-zone-universe-main-dev-postgresql-bouncer.fastorder.com, OVERRIDE_IP=10.100.1.78
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        pgbouncer
  Primary CN:  db-zone-universe-main-dev-postgresql-bouncer.fastorder.com
  Alt CN:      zone-universe-main-dev.fastorder.com
  Internal IP: 10.100.1.78 (from topology: db-coordinator-postgresql)
  Shared IP:   142.93.238.16 (legacy/public, also included in SANs)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 10:48:27 UTC] USER=www-data EUID=0 PID=878274 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer and /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:27 UTC] USER=www-data EUID=0 PID=878283 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer
🔐 Generating 4096-bit private key...
[2026-02-05 10:48:27 UTC] USER=www-data EUID=0 PID=878293 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-878237
[2026-02-05 10:48:27 UTC] USER=www-data EUID=0 PID=878302 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-878237/ra_root.crt
[2026-02-05 10:48:27 UTC] USER=www-data EUID=0 PID=878311 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-878237/ra_root.key
[2026-02-05 10:48:27 UTC] USER=www-data EUID=0 PID=878320 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-878237/ra_root.crt
[2026-02-05 10:48:27 UTC] USER=www-data EUID=0 PID=878329 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-878237/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-zone-universe-main-dev-postgresql-bouncer.fastorder.com
[2026-02-05 10:48:34 UTC] USER=www-data EUID=0 PID=878667 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-878237/server.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.key
[2026-02-05 10:48:34 UTC] USER=www-data EUID=0 PID=878676 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-878237/server.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.crt
[2026-02-05 10:48:34 UTC] USER=www-data EUID=0 PID=878685 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.crt
📋 Setting up CA certificate...
[2026-02-05 10:48:34 UTC] USER=www-data EUID=0 PID=878694 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-878237/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/root.crt
[2026-02-05 10:48:34 UTC] USER=www-data EUID=0 PID=878705 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/root.crt
[2026-02-05 10:48:34 UTC] USER=www-data EUID=0 PID=878736 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/root.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 10:48:34 UTC] USER=www-data EUID=0 PID=878765 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.key
[2026-02-05 10:48:34 UTC] USER=www-data EUID=0 PID=878775 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.key
[2026-02-05 10:48:34 UTC] USER=www-data EUID=0 PID=878784 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.crt
[2026-02-05 10:48:34 UTC] USER=www-data EUID=0 PID=878793 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.crt
[2026-02-05 10:48:34 UTC] USER=www-data EUID=0 PID=878806 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-zone-universe-main-dev-postgresql-bouncer.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-zone-universe-main-dev-postgresql-bouncer.fastorder.com, DNS:zone-universe-main-dev.fastorder.com, DNS:db-zone-universe-main-dev-postgresql-bouncer.fastorder.com, DNS:db-zone-universe-main-dev-postgresql-bouncer, DNS:localhost, IP Address:10.100.1.78, IP Address:127.0.0.1, IP Address:142.93.238.16
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: zone-universe-main-dev
Node:        pgbouncer
Primary CN:  db-zone-universe-main-dev-postgresql-bouncer.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/root.crt'

2. Restart PgBouncer:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@zone-universe-main-dev.service

3. Test SSL connection:
   psql "host=db-zone-universe-main-dev-postgresql-bouncer.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   mTLS server certificate present: /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.crt
[INFO] Generating pgbouncer_admin client certificates...
[INFO] ⏳ This may take 30-60 seconds...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Username:    pgbouncer_admin
Identifier:  pgbouncer
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: zone-universe-main-dev
  Service:     zone
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        pgbouncer
  User (CN):   pgbouncer_admin
  Hostname:    db-zone-universe-main-dev-postgresql-bouncer.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:48:35 UTC] USER=www-data EUID=0 PID=878974 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-pgbouncer-pgbouncer_admin
[2026-02-05 10:48:35 UTC] USER=www-data EUID=0 PID=879006 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-02-05 10:48:36 UTC] USER=www-data EUID=0 PID=879018 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = pgbouncer_admin
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:36 UTC] USER=www-data EUID=0 PID=879043 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:36 UTC] USER=www-data EUID=0 PID=879061 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 10:48:36 UTC] USER=www-data EUID=0 PID=879071 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 10:48:36 UTC] USER=www-data EUID=0 PID=879081 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/root.crt
[2026-02-05 10:48:36 UTC] USER=www-data EUID=0 PID=879096 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 10:48:36 UTC] USER=www-data EUID=0 PID=879107 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key.pkcs1 /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 10:48:36 UTC] USER=www-data EUID=0 PID=879117 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_der.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 10:48:37 UTC] USER=www-data EUID=0 PID=879126 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_pk8.der /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 10:48:37 UTC] USER=www-data EUID=0 PID=879136 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 10:48:37 UTC] USER=www-data EUID=0 PID=879145 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/root.crt
[2026-02-05 10:48:37 UTC] USER=www-data EUID=0 PID=879158 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:37 UTC] USER=www-data EUID=0 PID=879167 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 10:48:37 UTC] USER=www-data EUID=0 PID=879181 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 10:48:37 UTC] USER=www-data EUID=0 PID=879203 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 10:48:37 UTC] USER=www-data EUID=0 PID=879214 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 10:48:37 UTC] USER=www-data EUID=0 PID=879228 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/root.crt
[2026-02-05 10:48:37 UTC] USER=www-data EUID=0 PID=879241 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:37 UTC] USER=www-data EUID=0 PID=879277 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:37 UTC] USER=www-data EUID=0 PID=879286 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879317 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879333 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879342 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879351 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/root.crt
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879361 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/ca.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879372 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879386 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879418 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879431 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879443 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879465 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879474 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879485 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879495 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879504 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/pgbouncer/root.crt
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879519 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/ca.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879528 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/www-data/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879538 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879549 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/www-data/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 10:48:38 UTC] USER=www-data EUID=0 PID=879559 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/zone-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/zone-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879569 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879579 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879590 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879600 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879615 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879628 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879637 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879646 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/pgbouncer/root.crt
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879655 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/ca.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879671 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/postgres/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879681 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879706 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/postgres/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879720 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/zone-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/zone-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879742 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879753 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/zone-universe-main-dev
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879763 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879782 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879791 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879800 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/pgbouncer/root.crt
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879809 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/ca.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879818 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/kafka/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879827 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879836 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/kafka/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 10:48:39 UTC] USER=www-data EUID=0 PID=879846 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/zone-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/zone-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=879859 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=879868 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=879877 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=879886 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=879895 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=879904 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/zone-universe-main-dev
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=879923 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=879941 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=879950 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/zone-universe-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/zone-universe-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/zone-universe-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: zone-universe-main-dev
User: pgbouncer_admin
Node: pgbouncer
FQDN: db-zone-universe-main-dev-postgresql-bouncer.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/zone-universe-main-dev/pgbouncer/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/zone-universe-main-dev/pgbouncer/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-zone-universe-main-dev-postgresql-bouncer.fastorder.com -U pgbouncer_admin -d postgres

[OK]   mTLS client certificate present: /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[INFO] Creating symlinks to canonical certificates in /etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend...
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=879966 ACTION=fsop ARGS=mkdir -p /etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=879975 ACTION=fsop ARGS=mkdir -p /etc/ssl/private/zone-universe-main-dev/pg/pgbouncer-backend
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=879985 ACTION=fsop ARGS=ln -sf /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=879994 ACTION=fsop ARGS=ln -sf /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key /etc/ssl/private/zone-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=880003 ACTION=fsop ARGS=ln -sf /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/root.crt /etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/root.crt
[INFO] Creating coordinator CA symlink for PostgreSQL server verification...
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=880012 ACTION=fsop ARGS=ln -sf /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/root.crt /etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] Verifying canonical certificate permissions...
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=880021 ACTION=fsop ARGS=chmod 644 /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=880030 ACTION=fsop ARGS=chmod 640 /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=880040 ACTION=fsop ARGS=chmod 644 /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/root.crt
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=880049 ACTION=fsop ARGS=chown root:www-data /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key
[OK]   Backend certificate symlinks created in /etc/ssl
[OK]   Coordinator CA symlink created for server verification
[INFO] Creating symlinks in /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer for monitoring access...
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=880060 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer
[2026-02-05 10:48:40 UTC] USER=www-data EUID=0 PID=880069 ACTION=fsop ARGS=ln -sf /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt
/bin/ln: '/home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt' and '/etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt' are the same file
[2026-02-05 10:48:41 UTC] USER=www-data EUID=0 PID=880078 ACTION=fsop ARGS=ln -sf /home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key
/bin/ln: '/home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key' and '/etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key' are the same file
[OK]   Monitoring certificate symlinks created (or already exist)
[2026-02-05 10:48:41 UTC] USER=www-data EUID=0 PID=880088 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.crt
[2026-02-05 10:48:41 UTC] USER=www-data EUID=0 PID=880097 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.key
[2026-02-05 10:48:41 UTC] USER=www-data EUID=0 PID=880106 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 10:48:41 UTC] USER=www-data EUID=0 PID=880115 ACTION=fsop ARGS=test -r /etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] PgBouncer will use PostgreSQL coordinator CA: /etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[OK]   PostgreSQL coordinator at db-zone-universe-main-dev-postgresql-coordinator.fastorder.com:5432 is reachable
[INFO] Dumping SCRAM secrets from coordinator for PgBouncer auth_file …
[2026-02-05 10:48:41 UTC] USER=www-data EUID=0 PID=880134 ACTION=fsop ARGS=cp /tmp/tmp.crzLC2rjWt /etc/pgbouncer/zone-universe-main-dev/userlist.txt
[2026-02-05 10:48:41 UTC] USER=www-data EUID=0 PID=880143 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/zone-universe-main-dev/userlist.txt
[2026-02-05 10:48:41 UTC] USER=www-data EUID=0 PID=880152 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/zone-universe-main-dev/userlist.txt
[OK]   Auth file written: /etc/pgbouncer/zone-universe-main-dev/userlist.txt
[INFO] Generated new password for pgbouncer_admin
[INFO] Ensuring PgBouncer admin role 'pgbouncer_admin' exists in Postgres (coordinator) …
[OK]   Role pgbouncer_admin created/updated successfully
[SECRETS] Setting credentials in vault: fastorder/db/zone/universe/main/dev/postgresql/coordinator/pgbouncer_admin
✓ [SECRETS] Credentials created in vault: fastorder/db/zone/universe/main/dev/postgresql/coordinator/pgbouncer_admin
[INFO] ✅ PgBouncer admin password stored in centralized secrets vault
[INFO] Re-fetching SCRAM secrets after role creation to ensure pgbouncer_admin is included …
[2026-02-05 10:48:49 UTC] USER=www-data EUID=0 PID=880477 ACTION=fsop ARGS=cp /tmp/tmp.XEQbYTM9IE /etc/pgbouncer/zone-universe-main-dev/userlist.txt
[2026-02-05 10:48:49 UTC] USER=www-data EUID=0 PID=880502 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/zone-universe-main-dev/userlist.txt
[2026-02-05 10:48:49 UTC] USER=www-data EUID=0 PID=880511 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/zone-universe-main-dev/userlist.txt
[OK]   Auth file updated with pgbouncer_admin SCRAM hash
[INFO] Auth file contains [2026-02-05 10:48:49 UTC] USER=www-data EUID=0 PID=880529 ACTION=passthru ARGS=bash -c wc -l < '/etc/pgbouncer/zone-universe-main-dev/userlist.txt'
4 user(s)
[OK]   Admin 'pgbouncer_admin' password generated and saved
[INFO] Configuring PostgreSQL to prevent Citus metadata sync hangs...
ALTER ROLE
[OK]   Disabled Citus metadata sync for pgbouncer_admin
[INFO] Verifying application database fastorder_zone_universe_main_dev_db exists...
[OK]   ✓ Database fastorder_zone_universe_main_dev_db exists
[INFO] Granting permissions to pgbouncer_admin on fastorder_zone_universe_main_dev_db...
GRANT
[OK]   ✓ Granted CONNECT on fastorder_zone_universe_main_dev_db to pgbouncer_admin
GRANT
[OK]   ✓ Granted USAGE on schema public to pgbouncer_admin
GRANT
[OK]   ✓ Granted SELECT on all tables to pgbouncer_admin
ALTER DATABASE
[OK]   Set synchronous_commit=local for fastorder_zone_universe_main_dev_db
[INFO] Ensuring pg_hba.conf entry for pgbouncer_admin …
[INFO] Adding pg_hba.conf entries for pgbouncer_admin with cert auth …
[OK]   pg_hba.conf updated and PostgreSQL configuration reloaded
[2026-02-05 10:48:51 UTC] USER=unknown EUID=33 PID=880601 ACTION=-u ARGS=postgres bash
ERROR: Invalid or unauthorized action: -u
[WARN] pg_hba.conf entry may not have loaded correctly
[INFO] Writing /etc/pgbouncer/zone-universe-main-dev/pgbouncer.ini …
[2026-02-05 10:48:52 UTC] USER=www-data EUID=0 PID=880668 ACTION=fsop ARGS=cp /tmp/tmp.x6U9fvfALm /etc/pgbouncer/zone-universe-main-dev/pgbouncer.ini
[2026-02-05 10:48:52 UTC] USER=www-data EUID=0 PID=880677 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/zone-universe-main-dev/pgbouncer.ini
[2026-02-05 10:48:52 UTC] USER=www-data EUID=0 PID=880687 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/zone-universe-main-dev/pgbouncer.ini
[2026-02-05 10:48:52 UTC] USER=www-data EUID=0 PID=880698 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbouncer/zone-universe-main-dev /run/pgbouncer/zone-universe-main-dev /var/log/pgbouncer/zone-universe-main-dev
[2026-02-05 10:48:52 UTC] USER=www-data EUID=0 PID=880707 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/zone-universe-main-dev/userlist.txt
[OK]   pgbouncer.ini ready
[INFO] Verifying TLS settings in pgbouncer.ini:
[2026-02-05 10:48:53 UTC] USER=www-data EUID=0 PID=880721 ACTION=fsop ARGS=grep -E (client_tls_sslmode|server_tls) /etc/pgbouncer/zone-universe-main-dev/pgbouncer.ini
client_tls_sslmode = verify-full
server_tls_sslmode = verify-full
server_tls_ca_file = /etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
server_tls_cert_file = /etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
server_tls_key_file  = /etc/ssl/private/zone-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[INFO] Verifying PgBouncer server certificate files:
[2026-02-05 10:48:53 UTC] USER=www-data EUID=0 PID=880731 ACTION=fsop ARGS=test -r /etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[OK]   Server cert readable by postgres: /etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-02-05 10:48:53 UTC] USER=www-data EUID=0 PID=880744 ACTION=fsop ARGS=test -r /etc/ssl/private/zone-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[OK]   Server key readable by postgres: /etc/ssl/private/zone-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[INFO] Verifying coordinator CA certificate:
[2026-02-05 10:48:53 UTC] USER=www-data EUID=0 PID=880754 ACTION=fsop ARGS=test -r /etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[OK]   Coordinator CA readable by postgres: /etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] Preflight: stopping any conflicting PgBouncer on 6432 …
[2026-02-05 10:48:53 UTC] USER=www-data EUID=0 PID=880765 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer.service
[2026-02-05 10:48:53 UTC] USER=www-data EUID=0 PID=880777 ACTION=passthru ARGS=systemctl stop pgbouncer@zone-universe-main-dev.service
Failed to stop pgbouncer@zone-universe-main-dev.service: Unit pgbouncer@zone-universe-main-dev.service not loaded.
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json?all=1": dial unix /var/run/docker.sock: connect: permission denied
[WARN] Killing existing pgbouncer processes: 171960
179449
[2026-02-05 10:48:54 UTC] USER=www-data EUID=0 PID=880804 ACTION=passthru ARGS=bash -c kill -9 171960
[2026-02-05 10:48:54 UTC] USER=www-data EUID=0 PID=880826 ACTION=passthru ARGS=bash -c kill -9 179449
[2026-02-05 10:48:56 UTC] USER=www-data EUID=0 PID=880949 ACTION=passthru ARGS=systemctl daemon-reload
[OK]   systemd unit installed: pgbouncer@zone-universe-main-dev.service
[INFO] Running pre-flight IP conflict check for 10.100.1.78:6432 …
[WARN] IP conflict checker not found at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/check-ip-conflicts.sh
[WARN] Skipping pre-flight check - conflicts may occur
[INFO] Starting PgBouncer (zone-universe-main-dev) …
[2026-02-05 10:48:57 UTC] USER=www-data EUID=0 PID=881069 ACTION=passthru ARGS=systemctl restart pgbouncer@zone-universe-main-dev.service
[2026-02-05 10:48:57 UTC] USER=www-data EUID=0 PID=881081 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer@zone-universe-main-dev.service
[OK]   Service ACTIVE
[INFO] Verifying auth_file before probing …
[INFO] Auth file contains 4 user(s)
[WARN] Auth file does NOT contain pgbouncer_admin entry - authentication will fail
[INFO] Probing admin console via SSL (psql to database 'pgbouncer') …
[INFO] Retrieved password from vault for admin console probe
[WARN] Admin console probe failed (see error below)
psql: error: connection to server at "10.100.1.78", port 6432 failed: root certificate file "/etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
[WARN] Troubleshooting:
[WARN]   1. Check auth_file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/zone-universe-main-dev/userlist.txt
[WARN]   2. Test with: PGPASSWORD='T65wOqIzLUGdAZzD8hW72lpY' psql -h 10.100.1.78 -p 6432 -U pgbouncer_admin -d pgbouncer
[WARN]   3. Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@zone-universe-main-dev.service -n 50

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Running Comprehensive PgBouncer Verification Tests
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Password extracted: T65wOqIzLU... (using postgres user certificates)

[INFO] Test 1/7: Admin Console - SHOW POOLS
[WARN] ✗ SHOW POOLS: FAILED
[WARN] Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@zone-universe-main-dev.service -n 50

[INFO] Test 2/7: Admin Console - SHOW VERSION
[WARN] ✗ SHOW VERSION: FAILED

[INFO] Test 3/7: Admin Console - SHOW STATS
[WARN] ✗ SHOW STATS: FAILED

[INFO] Test 4/7: Admin Console - SHOW DATABASES
[WARN] ✗ SHOW DATABASES: FAILED

[INFO] Test 5/7: Admin Console - SHOW CONFIG
[WARN] ✗ SHOW CONFIG: FAILED
psql   "host=db-zone-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_zone_universe_main_dev_db user=pgbouncer_admin password=T65wOqIzLUGdAZzD8hW72lpY    connect_timeout=5 sslmode=verify-full    sslrootcert=/home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/root.crt    sslcert=/home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.crt    sslkey=/home/ab/ssl/.postgresql/zone-universe-main-dev/pgbouncer/pgbouncer_admin.key"   --no-psqlrc -Atc 'SELECT version();'

[INFO] Test 6/7: Application Database - SELECT version()
[WARN] ✗ Application database query: FAILED (timeout or connection issue)
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[INFO] Test 7/8: Application Database - Connection Details
[WARN] ✗ Connection details: FAILED (timeout or connection issue)
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[INFO] Test 8/8: End-to-End Application Routing - Pool Verification
[INFO]   Running actual queries through PgBouncer to verify routing and pooling...
[WARN] ✗ End-to-end routing verification: FAILED - All 3 queries failed
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[WARN]    Otherwise check if database fastorder_zone_universe_main_dev_db exists and user pgbouncer_admin has permissions

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Verification Complete - Tests 1-5 PASSED (Admin console verified)
[WARN]   Tests 6-8 FAILED - Application database not accessible
[WARN]   This is expected if Citus is not set up yet
[WARN]   Run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[OK]   PgBouncer is up for zone-universe-main-dev

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Connection Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Password stored in: AWS Secrets Manager (fastorder/db/web/ksa/main/dev/postgresqlzone/universe/main/dev/coordinator-pgbouncer_admin)
Current password: T65wOqIzLUGdAZzD8hW72lpY

1. Admin Console (using IP address to avoid DNS/SSL issues):
   psql "host=10.100.1.78 port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/zone-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=T65wOqIzLUGdAZzD8hW72lpY sslmode=verify-full sslrootcert=/etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

2. Admin Console (using hostname):
   psql "host=db-zone-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/zone-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=T65wOqIzLUGdAZzD8hW72lpY sslmode=verify-full sslrootcert=/etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

3. Application Database:
   psql "host=db-zone-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_zone_universe_main_dev_db sslkey=/etc/ssl/private/zone-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=T65wOqIzLUGdAZzD8hW72lpY sslmode=verify-full sslrootcert=/etc/ssl/certs/zone-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

4. Using .pgpass file:
   echo "db-zone-universe-main-dev-postgresql-bouncer.fastorder.com:6432:*:pgbouncer_admin:T65wOqIzLUGdAZzD8hW72lpY" >> ~/.pgpass
   chmod 600 ~/.pgpass
   psql -h db-zone-universe-main-dev-postgresql-bouncer.fastorder.com -p 6432 -U pgbouncer_admin -d fastorder_zone_universe_main_dev_db

5. Retrieve password from vault:
   source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
   PGPASSWORD="$(get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password')" \
     psql -h 10.100.1.78 -p 6432 -U pgbouncer_admin -d pgbouncer -c "SHOW POOLS;"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Architecture
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  • Default db 'fastorder_zone_universe_main_dev_db' → Citus coordinator (db-zone-universe-main-dev-postgresql-coordinator.fastorder.com)
  • Worker access: 'fastorder_zone_universe_main_dev_db_worker_1', 'fastorder_zone_universe_main_dev_db_worker_2', … (if exist)
  • Client TLS: require (password auth) / verify-full (mTLS with certs)
  • Server TLS: verify-full (PgBouncer validates PostgreSQL certs)
  • Auth: SCRAM-SHA-256 via /etc/pgbouncer/zone-universe-main-dev/userlist.txt
  • Pool mode: transaction (stateless connections)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Management
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Service Status:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer@zone-universe-main-dev.service
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@zone-universe-main-dev.service

Logs:
  command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@zone-universe-main-dev.service -f
  /usr/local/bin/fastorder-provisioning-wrapper.sh tail -f /var/log/pgbouncer/zone-universe-main-dev/pgbouncer.log

Reload Config:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@zone-universe-main-dev.service

Restart:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@zone-universe-main-dev.service

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Files
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Config:        /etc/pgbouncer/zone-universe-main-dev/pgbouncer.ini
Auth file:     /etc/pgbouncer/zone-universe-main-dev/userlist.txt
Server cert:   /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.crt
Server key:    /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/server.key
CA cert:       /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer/ca.crt
PG CA:         /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt
Logs:          /var/log/pgbouncer/zone-universe-main-dev/pgbouncer.log

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Troubleshooting
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


If "SASL authentication failed":
  1. Check auth file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/zone-universe-main-dev/userlist.txt
  2. Verify pgbouncer_admin is present with SCRAM hash
  3. Get password from vault:
     source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
     get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password'
  4. Reload PgBouncer: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@zone-universe-main-dev.service

If "no pg_hba.conf entry":
  1. Check pg_hba.conf on coordinator
  2. Add rule: hostssl all pgbouncer_admin 10.100.1.78/32 cert clientcert=verify-full
  3. Reload PostgreSQL

To add users to PgBouncer:
  1. Create user in PostgreSQL with password
  2. Re-run SCRAM dump:
     psql "host=db-zone-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 dbname=postgres user=postgres \
       sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/root.crt \
       sslcert=/etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.crt sslkey=/etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key" \
       -Atc "SELECT '\"' || rolname || '\" \"' || rolpassword || '\"' \
             FROM pg_authid WHERE rolpassword LIKE 'SCRAM-SHA-256%' \
             AND rolcanlogin ORDER BY rolname;" | command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop tee /etc/pgbouncer/zone-universe-main-dev/userlist.txt
  3. Reload: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@zone-universe-main-dev.service

[INFO] Registering PgBouncer node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        zone-universe-main-dev-pgbouncer
[INFO]   Identifier Parent: postgresql
[INFO]   IP:                10.100.1.78
[INFO]   Port:              6432
[INFO]   FQDN:              db-zone-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 80d2b3fc-6c2e-432b-ab5b-883cbc58f827
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   PgBouncer node registered to observability API
✓ ✅ PgBouncer setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 03-citus-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] CITUS DISTRIBUTED CLUSTER SETUP
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Phase 1: Installing Citus extension on workers...
[INFO] Phase 2: Setting up coordinator and registering workers...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 📦 PHASE 1: Installing Citus extension on 1 worker(s)...

[INFO] → Worker 1/1: Installing Citus on worker-01...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS CLUSTER SETUP
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔧 Setting up Citus Worker...
[INFO] Temporarily disabling synchronous replication for extension installation...
t
[INFO] Installing Citus extension on worker...
[OK]   Citus extension installed on worker
[INFO] Restoring synchronous replication settings...
t
[INFO] Worker Citus extension installed - registration will happen when coordinator setup runs

[OK]   Citus setup complete for worker-01
[INFO] ═══════════════════════════════════════════════════════════════════════════════
✓   ✅ Citus extension installed on worker-01

✓ ✅ Phase 1 Complete: All 1 workers have Citus extension installed

[INFO] 🔧 PHASE 2: Setting up Citus coordinator and registering workers...

[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS CLUSTER SETUP
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔧 Setting up Citus Coordinator...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] DIAGNOSTIC: Configuration Variables
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PG_WORKERS_NUM: 1
[INFO] ENV_ID: zone-universe-main-dev
[INFO] DOMAIN: fastorder.com
[INFO] PORT: 5432
[INFO] SOCKET_DIR: /var/run/postgresql-zone-universe-main-dev-coordinator
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Ensuring postgres client certificates exist for coordinator...
[OK]   Postgres client certificates already exist for coordinator
[INFO] Adding citus_cert_map to coordinator pg_ident.conf...
[OK]   pg_ident.conf updated for coordinator
[INFO] Installing Citus extension on coordinator...
[OK]   Citus extension installed on coordinator (postgres database)
[INFO] Installing Citus extension on application database: fastorder_zone_universe_main_dev_db...
[OK]   Citus extension installed on application database: fastorder_zone_universe_main_dev_db
[INFO] Configuring Citus SSL connection parameters...
[2026-02-05 10:49:12 UTC] USER=www-data EUID=0 PID=882028 ACTION=passthru ARGS=systemctl reload postgresql@zone-universe-main-dev-coordinator.service
[OK]   ✅ Citus SSL connection parameters configured: /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator
[WARN] Node not identified as coordinator, initializing...
[INFO] Checking coordinator configuration...
[INFO] Persisting citus.local_hostname to postgresql.conf...
[2026-02-05 10:49:14 UTC] USER=www-data EUID=0 PID=882106 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/zone-universe-main-dev/coordinator/postgresql.conf
[2026-02-05 10:49:14 UTC] USER=www-data EUID=0 PID=882133 ACTION=passthru ARGS=systemctl reload postgresql@zone-universe-main-dev-coordinator.service
[OK]   ✅ citus.local_hostname persisted to config and reloaded
[INFO] Configuring coordinator hostname in postgres database: db-zone-universe-main-dev-postgresql-coordinator.fastorder.com:5432

[OK]   ✅ Coordinator hostname set to db-zone-universe-main-dev-postgresql-coordinator.fastorder.com:5432 in postgres database
[INFO] Checking coordinator configuration in application database: fastorder_zone_universe_main_dev_db...
[WARN] ⚠️  Coordinator registered as 'localhost' in application database, fixing...
[INFO] Configuring coordinator hostname in application database: db-zone-universe-main-dev-postgresql-coordinator.fastorder.com:5432
[OK]   ✅ Coordinator hostname set to db-zone-universe-main-dev-postgresql-coordinator.fastorder.com:5432 in application database
[INFO] Validating coordinator configuration before worker registration...
[OK]   ✅ Coordinator hostname validated: db-zone-universe-main-dev-postgresql-coordinator.fastorder.com
[OK]   ✅ citus_tables view is accessible
[INFO] Checking coordinator self-registration...
[OK]   ✅ Coordinator is already self-registered
[INFO] Configuring coordinator shard placement policy...
[OK]   ✅ Coordinator already configured in postgres database (shouldhaveshards = false)
[WARN] ⚠️  Coordinator has 66 shards in fastorder_zone_universe_main_dev_db - cannot set shouldhaveshards=false
[WARN]    You must rebalance shards to workers first, then run this setup again
[WARN]    Skipping shouldhaveshards configuration for application database
[INFO] Registering 1 worker(s) to Citus cluster...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PRE-FLIGHT: Checking worker availability...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking worker worker-01...
[INFO]   FQDN: db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
[OK]   ✅ Worker worker-01 is reachable via SSL
[OK]   All workers are reachable - proceeding with registration

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Adding Citus worker: db-zone-universe-main-dev-postgresql-worker-01.fastorder.com:5432
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Adding citus_cert_map to worker-01 pg_ident.conf...
[OK]   pg_ident.conf updated for worker-01
[INFO] Configuring worker worker-01 HBA for coordinator (10.100.1.55) access...
[OK]   Worker worker-01 HBA configured for coordinator (10.100.1.55)
[INFO] Adding replication rules for 3 standby(s)...
[OK]   Replication rules already exist for worker-01
[INFO] Reloading worker worker-01 to apply HBA changes...
[2026-02-05 10:49:18 UTC] USER=www-data EUID=0 PID=882393 ACTION=passthru ARGS=systemctl reload postgresql@zone-universe-main-dev-worker-01.service
[INFO] Configuring coordinator HBA for worker worker-01 (10.100.1.106) access...
[OK]   Coordinator HBA configured for worker worker-01 (10.100.1.106)
[INFO] Reloading coordinator to apply HBA changes...
[2026-02-05 10:49:18 UTC] USER=www-data EUID=0 PID=882425 ACTION=passthru ARGS=systemctl reload postgresql@zone-universe-main-dev-coordinator.service
[INFO] Ensuring postgres client certificates exist for worker-01...
[OK]   Postgres client certificates already exist for worker-01
[INFO] Configuring citus.node_conninfo on worker-01...
[2026-02-05 10:49:18 UTC] USER=www-data EUID=0 PID=882452 ACTION=passthru ARGS=systemctl reload postgresql@zone-universe-main-dev-worker-01.service
[OK]   citus.node_conninfo configured on worker-01
[INFO] Temporarily relaxing sync-rep on worker worker-01...
t
[OK]   Worker worker-01 sync-rep relaxed (was: sync_commit=on)
[INFO] Ensuring Citus extension on worker databases...
CREATE EXTENSION
CREATE EXTENSION
[INFO] Running citus_add_node with 180s timeout...
NOTICE:  shards are still on the coordinator after adding the new node
HINT:  Use SELECT rebalance_table_shards(); to balance shards data between workers and coordinator or SELECT citus_drain_node('db-zone-universe-main-dev-postgresql-coordinator.fastorder.com',5432); to permanently move shards away from the coordinator.
2
[INFO] Restoring worker worker-01 sync-rep settings...
t
[OK]   Worker worker-01 sync-rep restored
[OK]   ✅ Worker db-zone-universe-main-dev-postgresql-worker-01.fastorder.com successfully added to Citus cluster
[INFO]    Node ID: 2
[INFO]    Registered in: postgres, fastorder_zone_universe_main_dev_db
[OK]   Worker worker-01 registration successful
[INFO] Configuring worker worker-01 shard placement policy...
[OK]   ✅ Worker worker-01 configured to hold shards in all databases


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] POST-REGISTRATION: Verifying cluster state...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Expected workers: 1
[INFO] Registered workers: 1
[OK]   ✅ All 1 workers successfully registered!

[INFO] Citus cluster configuration:
db-zone-universe-main-dev-postgresql-coordinator.fastorder.com  5432  0  t  primary  f
db-zone-universe-main-dev-postgresql-worker-01.fastorder.com    5432  1  t  primary  t

[INFO] Note: groupid=0 is the coordinator, groupid>0 are workers
[INFO]       shouldhaveshards: false=query router only, true=holds data shards

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] FINAL VALIDATION: Verifying configuration persistence...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:49:22 UTC] USER=www-data EUID=0 PID=882799 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/zone-universe-main-dev/coordinator/postgresql.conf
[OK]   ✅ citus.local_hostname persisted in postgresql.conf
[OK]   ✅ All 1 worker(s) successfully registered and verified

[OK]   ✅ All validation checks passed
[OK]   Citus coordinator setup complete

[OK]   Citus setup complete for coordinator
[INFO] ═══════════════════════════════════════════════════════════════════════════════

✓ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ CITUS CLUSTER SETUP COMPLETED SUCCESSFULLY
✓    Coordinator: Ready and accepting connections
✓    Workers registered: 1
✓ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 05-backup-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up coordinator backup...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for zone-universe-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-02-05 10:49:25 UTC] USER=www-data EUID=0 PID=882991 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/zone-universe-main-dev
[2026-02-05 10:49:25 UTC] USER=www-data EUID=0 PID=883009 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/zone-universe-main-dev
[2026-02-05 10:49:25 UTC] USER=www-data EUID=0 PID=883030 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-02-05 10:49:25 UTC] USER=www-data EUID=0 PID=883060 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-02-05 10:49:25 UTC] USER=www-data EUID=0 PID=883069 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-02-05 10:49:36 UTC] USER=www-data EUID=0 PID=883523 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-02-05 10:49:36 UTC] USER=www-data EUID=0 PID=883532 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-02-05 10:49:36 UTC] USER=www-data EUID=0 PID=883541 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-02-05 10:49:36 UTC] USER=www-data EUID=0 PID=883550 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/zone-universe-main-dev
[2026-02-05 10:49:36 UTC] USER=www-data EUID=0 PID=883559 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/zone-universe-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Generating new cipher key and saving to /etc/pgbackrest/.cipher-key-zone-universe-main-dev
[2026-02-05 10:49:36 UTC] USER=www-data EUID=0 PID=883581 ACTION=fsop ARGS=chmod 600 /etc/pgbackrest/.cipher-key-zone-universe-main-dev
[2026-02-05 10:49:36 UTC] USER=www-data EUID=0 PID=883590 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/.cipher-key-zone-universe-main-dev
[2026-02-05 10:49:36 UTC] USER=www-data EUID=0 PID=883611 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-02-05 10:49:36 UTC] USER=www-data EUID=0 PID=883621 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-02-05 10:49:36 UTC] USER=www-data EUID=0 PID=883630 ACTION=fsop ARGS=find /data/postgresql/17/zone-universe-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-02-05 10:49:37 UTC] USER=www-data EUID=0 PID=883639 ACTION=fsop ARGS=chown -R postgres:postgres /data/postgresql/17/zone-universe-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-02-05 10:49:37 UTC] USER=www-data EUID=0 PID=883653 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-02-05 10:49:37 UTC] USER=www-data EUID=0 PID=883668 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-02-05 10:49:37 UTC] USER=www-data EUID=0 PID=883677 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-02-05 10:49:37 UTC] USER=www-data EUID=0 PID=883686 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/zone-universe-main-dev/coordinator/PG_VERSION
[2026-02-05 10:49:37 UTC] USER=www-data EUID=0 PID=883697 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@zone-universe-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[2026-02-05 10:49:37 UTC] USER=www-data EUID=0 PID=883735 ACTION=fsop ARGS=rm -rf /var/lib/pgbackrest/backup/zone-universe-main-dev/backup/zone-universe-main-dev-coordinator
[INFO] Creating stanza: zone-universe-main-dev-coordinator...
2026-02-05 10:49:37.621 P00   INFO: stanza-create command begin 2.56.0: --exec-id=883752-56c92830 --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/zone-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-zone-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/zone-universe-main-dev --stanza=zone-universe-main-dev-coordinator
2026-02-05 10:49:37.643 P00   INFO: stanza-create for stanza 'zone-universe-main-dev-coordinator' on repo1
2026-02-05 10:49:37.656 P00   INFO: stanza-create command end: completed successfully (40ms)
[INFO] ✅ Stanza created successfully

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-02-05 10:49:37 UTC] USER=www-data EUID=0 PID=883771 ACTION=passthru ARGS=systemctl stop postgresql@zone-universe-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-02-05 10:49:40 UTC] USER=www-data EUID=0 PID=883808 ACTION=passthru ARGS=systemctl start postgresql@zone-universe-main-dev-coordinator.service
[2026-02-05 10:49:44 UTC] USER=www-data EUID=0 PID=884089 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@zone-universe-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-02-05 10:49:44 UTC] USER=www-data EUID=0 PID=884111 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=zone-universe-main-dev-coordinator --log-level-console=info check
2026-02-05 10:49:45.025 P00   INFO: check command begin 2.56.0: --exec-id=884118-6cd9a119 --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/zone-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-zone-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/zone-universe-main-dev --stanza=zone-universe-main-dev-coordinator
2026-02-05 10:49:45.052 P00   INFO: check repo1 configuration (primary)
2026-02-05 10:49:45.116 P00   INFO: check repo1 archive for WAL (primary)
2026-02-05 10:49:45.719 P00   INFO: WAL segment 000000010000000000000004 successfully archived to '/var/lib/pgbackrest/backup/zone-universe-main-dev/archive/zone-universe-main-dev-coordinator/17-1/0000000100000000/000000010000000000000004-6b6fd9416ff7b30112335436c8e3ee8a2ff8a25f.lz4' on repo1
2026-02-05 10:49:45.719 P00   INFO: check command end: completed successfully (701ms)
[INFO] ✅ Stanza verification passed

[INFO] 8️⃣ Creating backup automation scripts...
[2026-02-05 10:49:45 UTC] USER=www-data EUID=0 PID=884167 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|zone-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-zone-universe-main-dev.sh
[2026-02-05 10:49:45 UTC] USER=www-data EUID=0 PID=884176 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-zone-universe-main-dev.sh
[2026-02-05 10:49:45 UTC] USER=www-data EUID=0 PID=884194 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|zone-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-zone-universe-main-dev.sh
[2026-02-05 10:49:46 UTC] USER=www-data EUID=0 PID=884203 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-zone-universe-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-02-05 10:49:46 UTC] USER=www-data EUID=0 PID=884283 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/data/postgresql/17/zone-universe-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_zone-universe-main-dev.md
[2026-02-05 10:49:46 UTC] USER=www-data EUID=0 PID=884292 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_zone-universe-main-dev.md
[2026-02-05 10:49:46 UTC] USER=www-data EUID=0 PID=884301 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_zone-universe-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_zone-universe-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-02-05 10:49:46.878 P00   INFO: start command begin 2.56.0: --exec-id=884336-1502cb04 --log-level-console=info --log-level-file=debug --stanza=zone-universe-main-dev-coordinator
2026-02-05 10:49:46.878 P00   WARN: stop file does not exist for stanza zone-universe-main-dev-coordinator
2026-02-05 10:49:46.878 P00   INFO: start command end: completed successfully (8ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-02-05 10:49:47.011 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=884354-59f43ba2 --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/zone-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-zone-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/zone-universe-main-dev --stanza=zone-universe-main-dev-coordinator
2026-02-05 10:49:47.014 P00   INFO: stanza-upgrade for stanza 'zone-universe-main-dev-coordinator' on repo1
2026-02-05 10:49:47.018 P00   INFO: stanza 'zone-universe-main-dev-coordinator' on repo1 is already up to date
2026-02-05 10:49:47.018 P00   INFO: stanza-upgrade command end: completed successfully (23ms)
[INFO] This may take a few minutes depending on database size...
[2026-02-05 10:49:47 UTC] USER=www-data EUID=0 PID=884359 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260205-104947.log
[2026-02-05 10:49:47 UTC] USER=www-data EUID=0 PID=884368 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260205-104947.log
[2026-02-05 10:49:47 UTC] USER=www-data EUID=0 PID=884379 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260205-104947.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-02-05 10:50:02 UTC] USER=www-data EUID=0 PID=884951 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-882948.log /var/log/pgbackrest/initial-backup-20260205-104947.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260205-104947.log
   2026-02-05 10:50:01.732 P00   INFO: new backup label = 20260205-104947F
   2026-02-05 10:50:02.055 P00   INFO: full backup size = 55.8MB, file total = 1970
   2026-02-05 10:50:02.056 P00   INFO: backup command end: completed successfully (14718ms)
   2026-02-05 10:50:02.056 P00   INFO: expire command begin 2.56.0: --exec-id=884396-aad8e82b --log-level-console=info --log-level-file=debug --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/zone-universe-main-dev --repo1-retention-diff=4 --repo1-retention-full=4 --stanza=zone-universe-main-dev-coordinator
   2026-02-05 10:50:02.060 P00   INFO: expire command end: completed successfully (4ms)

[INFO] Current backups:
stanza: zone-universe-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (current)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000005

        full backup: 20260205-104947F
            timestamp start/stop: 2026-02-05 10:49:47+00 / 2026-02-05 10:50:01+00
            wal start/stop: 000000010000000000000005 / 000000010000000000000005
            database size: 55.8MB, database backup size: 55.8MB
            repo1: backup set size: 7.9MB, backup size: 7.9MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         zone-universe-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/zone-universe-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/zone-universe-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_zone-universe-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=zone-universe-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up worker backups for 1 worker(s)...
[INFO] Setting up backup for: worker-01
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for zone-universe-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-02-05 10:50:04 UTC] USER=www-data EUID=0 PID=885194 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/zone-universe-main-dev
[2026-02-05 10:50:05 UTC] USER=www-data EUID=0 PID=885218 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-02-05 10:50:05 UTC] USER=www-data EUID=0 PID=885245 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-02-05 10:50:05 UTC] USER=www-data EUID=0 PID=885256 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-02-05 10:50:12 UTC] USER=www-data EUID=0 PID=885544 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-02-05 10:50:12 UTC] USER=www-data EUID=0 PID=885556 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-02-05 10:50:12 UTC] USER=www-data EUID=0 PID=885577 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-02-05 10:50:12 UTC] USER=www-data EUID=0 PID=885594 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/zone-universe-main-dev
[2026-02-05 10:50:12 UTC] USER=www-data EUID=0 PID=885604 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/zone-universe-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-zone-universe-main-dev
[2026-02-05 10:50:12 UTC] USER=www-data EUID=0 PID=885632 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-02-05 10:50:12 UTC] USER=www-data EUID=0 PID=885641 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-02-05 10:50:12 UTC] USER=www-data EUID=0 PID=885650 ACTION=fsop ARGS=find /data/postgresql/17/zone-universe-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-02-05 10:50:12 UTC] USER=www-data EUID=0 PID=885660 ACTION=fsop ARGS=chown -R postgres:postgres /data/postgresql/17/zone-universe-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-02-05 10:50:13 UTC] USER=www-data EUID=0 PID=885670 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-02-05 10:50:13 UTC] USER=www-data EUID=0 PID=885679 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-02-05 10:50:13 UTC] USER=www-data EUID=0 PID=885689 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-02-05 10:50:13 UTC] USER=www-data EUID=0 PID=885698 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/zone-universe-main-dev/coordinator/PG_VERSION
[2026-02-05 10:50:13 UTC] USER=www-data EUID=0 PID=885708 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@zone-universe-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza zone-universe-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-02-05 10:50:14 UTC] USER=www-data EUID=0 PID=885808 ACTION=passthru ARGS=systemctl stop postgresql@zone-universe-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-02-05 10:50:16 UTC] USER=www-data EUID=0 PID=885947 ACTION=passthru ARGS=systemctl start postgresql@zone-universe-main-dev-coordinator.service
[2026-02-05 10:50:20 UTC] USER=www-data EUID=0 PID=886154 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@zone-universe-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-02-05 10:50:20 UTC] USER=www-data EUID=0 PID=886178 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=zone-universe-main-dev-coordinator --log-level-console=info check
2026-02-05 10:50:20.690 P00   INFO: check command begin 2.56.0: --exec-id=886186-e28609f3 --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/zone-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-zone-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/zone-universe-main-dev --stanza=zone-universe-main-dev-coordinator
2026-02-05 10:50:20.750 P00   INFO: check repo1 configuration (primary)
2026-02-05 10:50:20.801 P00   INFO: check repo1 archive for WAL (primary)
2026-02-05 10:50:21.403 P00   INFO: WAL segment 000000010000000000000007 successfully archived to '/var/lib/pgbackrest/backup/zone-universe-main-dev/archive/zone-universe-main-dev-coordinator/17-1/0000000100000000/000000010000000000000007-5a2ed95095c2ee5f44164860dc809caad6d1d4fe.lz4' on repo1
2026-02-05 10:50:21.403 P00   INFO: check command end: completed successfully (717ms)
[INFO] ✅ Stanza verification passed

[INFO] 8️⃣ Creating backup automation scripts...
[2026-02-05 10:50:21 UTC] USER=www-data EUID=0 PID=886219 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|zone-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-zone-universe-main-dev.sh
[2026-02-05 10:50:21 UTC] USER=www-data EUID=0 PID=886231 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-zone-universe-main-dev.sh
[2026-02-05 10:50:21 UTC] USER=www-data EUID=0 PID=886251 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|zone-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-zone-universe-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-02-05 10:50:22 UTC] USER=www-data EUID=0 PID=886340 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-zone-universe-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-02-05 10:50:22 UTC] USER=www-data EUID=0 PID=886358 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|zone-universe-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_zone-universe-main-dev.md
[2026-02-05 10:50:22 UTC] USER=www-data EUID=0 PID=886368 ACTION=fsop ARGS=sed -i s|__ENV_ID__|zone-universe-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_zone-universe-main-dev.md
[2026-02-05 10:50:22 UTC] USER=www-data EUID=0 PID=886379 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/data/postgresql/17/zone-universe-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_zone-universe-main-dev.md
[2026-02-05 10:50:22 UTC] USER=www-data EUID=0 PID=886388 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_zone-universe-main-dev.md
[2026-02-05 10:50:22 UTC] USER=www-data EUID=0 PID=886401 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_zone-universe-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_zone-universe-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-02-05 10:50:23.089 P00   INFO: start command begin 2.56.0: --exec-id=886430-680f099d --log-level-console=info --log-level-file=debug --stanza=zone-universe-main-dev-coordinator
2026-02-05 10:50:23.092 P00   WARN: stop file does not exist for stanza zone-universe-main-dev-coordinator
2026-02-05 10:50:23.092 P00   INFO: start command end: completed successfully (19ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-02-05 10:50:23.155 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=886441-a41f65c0 --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/zone-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-zone-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/zone-universe-main-dev --stanza=zone-universe-main-dev-coordinator
2026-02-05 10:50:23.157 P00   INFO: stanza-upgrade for stanza 'zone-universe-main-dev-coordinator' on repo1
2026-02-05 10:50:23.159 P00   INFO: stanza 'zone-universe-main-dev-coordinator' on repo1 is already up to date
2026-02-05 10:50:23.159 P00   INFO: stanza-upgrade command end: completed successfully (8ms)
[INFO] This may take a few minutes depending on database size...
[2026-02-05 10:50:23 UTC] USER=www-data EUID=0 PID=886445 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260205-105023.log
[2026-02-05 10:50:23 UTC] USER=www-data EUID=0 PID=886454 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260205-105023.log
[2026-02-05 10:50:23 UTC] USER=www-data EUID=0 PID=886470 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260205-105023.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-02-05 10:50:29 UTC] USER=www-data EUID=0 PID=886714 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-885099.log /var/log/pgbackrest/initial-backup-20260205-105023.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260205-105023.log
   2026-02-05 10:50:29.362 P00   INFO: new backup label = 20260205-105023F
   2026-02-05 10:50:29.596 P00   INFO: full backup size = 55.8MB, file total = 1970
   2026-02-05 10:50:29.597 P00   INFO: backup command end: completed successfully (6011ms)
   2026-02-05 10:50:29.597 P00   INFO: expire command begin 2.56.0: --exec-id=886493-8aee7025 --log-level-console=info --log-level-file=debug --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/zone-universe-main-dev --repo1-retention-diff=4 --repo1-retention-full=4 --stanza=zone-universe-main-dev-coordinator
   2026-02-05 10:50:29.600 P00   INFO: expire command end: completed successfully (3ms)

[INFO] Current backups:
stanza: zone-universe-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (current)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000008

        full backup: 20260205-104947F
            timestamp start/stop: 2026-02-05 10:49:47+00 / 2026-02-05 10:50:01+00
            wal start/stop: 000000010000000000000005 / 000000010000000000000005
            database size: 55.8MB, database backup size: 55.8MB
            repo1: backup set size: 7.9MB, backup size: 7.9MB

        full backup: 20260205-105023F
            timestamp start/stop: 2026-02-05 10:50:23+00 / 2026-02-05 10:50:29+00
            wal start/stop: 000000010000000000000008 / 000000010000000000000008
            database size: 55.8MB, database backup size: 55.8MB
            repo1: backup set size: 7.9MB, backup size: 7.9MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         zone-universe-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/zone-universe-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/zone-universe-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_zone-universe-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=zone-universe-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ Backup setup completed for coordinator and all workers

[INFO] Skipping 06-distribute-tables-canary.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 07-distribute-tables.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:50:31 UTC] USER=unknown EUID=33 PID=886813 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 10:50:31 UTC] USER=unknown EUID=33 PID=886820 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 10:50:31 UTC] USER=unknown EUID=33 PID=886830 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 10:50:31 UTC] USER=unknown EUID=33 PID=886844 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS TABLE DISTRIBUTION
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔐 Secure connection established
[INFO]    Host: db-zone-universe-main-dev-postgresql-coordinator.fastorder.com:5432
[INFO]    Database: fastorder_zone_universe_main_dev_db
[INFO]    SSL: verify-full (TLS 1.2+)
[INFO]    Timeouts: statement=120s, idle_tx=300s

[INFO] 🔍 Running preflight checks...
[INFO] Testing database connectivity...
[OK]   ✅ Database connection successful
[OK]   ✅ Connected to correct database: fastorder_zone_universe_main_dev_db
[INFO] Checking Citus extension in database fastorder_zone_universe_main_dev_db...
[OK]   Citus version: 13.2-1
[INFO] Checking worker registration...
[OK]   Registered workers: 1
[INFO] Worker nodes:
[INFO]                              nodename                           | nodeport | isactive | noderole 
[INFO]   --------------------------------------------------------------+----------+----------+----------
[INFO]    db-zone-universe-main-dev-postgresql-worker-01.fastorder.com |     5432 | t        | primary
[INFO]   (1 row)
[INFO]   

[INFO] 📊 Starting table distribution...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Distributing: auth.login_account
[INFO] Description: User authentication table - distributed by region for tenant isolation
[INFO] Shard key: region_hint
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ⏭️  Table does not exist, skipping

[INFO] ═══════════════════════════════════════════════════════════════════════════════
[OK]   ✅ All tables distributed successfully!
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 📊 Citus Cluster Summary:

[INFO] Distributed tables:
[INFO]             table          |    type     | shard_key | shards |  size   
[INFO]   -------------------------+-------------+-----------+--------+---------
[INFO]    zone.contract_key       | reference   | <none>    |      1 | 16 kB
[INFO]    zone.contract_type      | reference   | <none>    |      1 | 16 kB
[INFO]    zone.contract_term_json | distributed | id        |     32 | 512 kB
[INFO]    zone.contract_term_vars | distributed | id        |     32 | 1792 kB
[INFO]   (4 rows)
[INFO]   

[INFO] Worker capacity:
[INFO]    worker | total_shards | total_size 
[INFO]   --------+--------------+------------
[INFO]   (0 rows)
[INFO]   

[OK]   Citus table distribution complete

[INFO] Skipping 08-distribute-tables-rollback.sh (rollback script - run manually only)
[INFO] Skipping 09-distribute-tables-test.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 10-setup-cdc.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] CDC PIPELINE SETUP (Debezium + Elasticsearch Sink)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Log file: /var/log/fastorder/cdc/10-setup-cdc-*.log

[INFO] Running CDC setup for identifier: coordinator
[2026-02-05 10:50:39] ==========================================
[2026-02-05 10:50:39] CDC SETUP SCRIPT STARTED
[2026-02-05 10:50:39] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260205_105039.log
[2026-02-05 10:50:39] ==========================================
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[2026-02-05 10:50:40] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:50:40]   CDC Pipeline Setup (Debezium + ES Sink)
[2026-02-05 10:50:40] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:50:40]   Environment: zone-universe-main-dev
[2026-02-05 10:50:40]   Identifier:  coordinator
[2026-02-05 10:50:40]   Service:     zone
[2026-02-05 10:50:40] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 10:50:40] 📂 CDC_BASE_DIR exists: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc
[2026-02-05 10:50:40] Looking for service folder: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/zone
[2026-02-05 10:50:40] ⚠️  No CDC configuration for service 'zone' at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/zone
[2026-02-05 10:50:40] ℹ️  Skipping CDC setup for this service
✓ ✅ CDC Pipeline setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 11-monitoring-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up monitoring for coordinator...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 PostgreSQL Monitoring Integration for zone-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-zone-universe-main-dev
[OK]   Observability cell endpoints registered for zone-universe-main-dev
[OK]   ✓ Observability cell is ready

[INFO] ✓ Using private IP for metrics: 10.100.1.55
[INFO] 2️⃣ Setting up postgres_exporter integration...
[INFO] Checking observability cell readiness: obs-zone-universe-main-dev
[OK]   Observability cell endpoints registered for zone-universe-main-dev
[INFO] Setting up postgres_exporter for zone-universe-main-dev
[2026-02-05 10:50:43 UTC] USER=www-data EUID=0 PID=887715 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-zone-universe-main-dev.yaml /etc/prometheus/postgres_exporter_queries-zone-universe-main-dev.yaml
[2026-02-05 10:50:43 UTC] USER=www-data EUID=0 PID=887729 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-zone-universe-main-dev.yaml
[2026-02-05 10:50:43 UTC] USER=www-data EUID=0 PID=887741 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-zone-universe-main-dev.yaml
[OK]   Custom queries file created at /etc/prometheus/postgres_exporter_queries-zone-universe-main-dev.yaml
[2026-02-05 10:50:44 UTC] USER=www-data EUID=0 PID=887751 ACTION=passthru ARGS=mv /tmp/postgres_exporter-zone-universe-main-dev.service /etc/systemd/system/postgres_exporter-zone-universe-main-dev.service
[2026-02-05 10:50:44 UTC] USER=www-data EUID=0 PID=887760 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 10:50:45 UTC] USER=www-data EUID=0 PID=887818 ACTION=passthru ARGS=systemctl enable postgres_exporter-zone-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/postgres_exporter-zone-universe-main-dev.service → /etc/systemd/system/postgres_exporter-zone-universe-main-dev.service.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  IP Conflict Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: zone-universe-main-dev
IP Address:  10.100.1.55
Port:        9187
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔍 Checking IP conflict for zone-universe-main-dev on 10.100.1.55:9187...
✅ IP 10.100.1.55:9187 is available - no conflicts detected

🔍 Checking for orphaned processes that might conflict...
✅ No orphaned processes detected

✅ All checks passed - safe to proceed with zone-universe-main-dev setup
[2026-02-05 10:50:47 UTC] USER=www-data EUID=0 PID=888013 ACTION=passthru ARGS=systemctl restart postgres_exporter-zone-universe-main-dev.service
[OK]   postgres_exporter configured on db-zone-universe-main-dev-postgresql.fastorder.com:9187
[INFO] Adding PostgreSQL scrape target to Prometheus config...
[OK]   PostgreSQL scrape target added
[INFO] Creating PostgreSQL alert rules...
[2026-02-05 10:50:49 UTC] USER=www-data EUID=0 PID=888114 ACTION=fsop ARGS=mv /tmp/postgresql_alerts_zone-universe-main-dev.yml /etc/prometheus/obs-zone-universe-main-dev/rules/postgresql_alerts.yml
[OK]   PostgreSQL alert rules created: /etc/prometheus/obs-zone-universe-main-dev/rules/postgresql_alerts.yml
[INFO] Adding PostgreSQL alerts to Prometheus config...
[2026-02-05 10:50:49 UTC] USER=www-data EUID=0 PID=888124 ACTION=fsop ARGS=sed -i /rule_files:/a\  - "rules/postgresql_alerts.yml" /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml
[OK]   PostgreSQL alerts registered in Prometheus
[2026-02-05 10:50:49 UTC] USER=www-data EUID=0 PID=888134 ACTION=passthru ARGS=systemctl reload prometheus-obs-zone-universe-main-dev.service
Failed to reload prometheus-obs-zone-universe-main-dev.service: Job type reload is not applicable for unit prometheus-obs-zone-universe-main-dev.service.
[2026-02-05 10:50:49 UTC] USER=www-data EUID=0 PID=888143 ACTION=passthru ARGS=systemctl restart prometheus-obs-zone-universe-main-dev.service
[OK]   Prometheus reloaded with PostgreSQL monitoring
[OK]   ✓ postgres_exporter integration complete
[INFO] Registering postgres_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: postgres_exporter -> 10.100.1.55:9187
[OK]   ✓ Registered postgres_exporter scrape target: 10.100.1.55:9187
[INFO]   Target file: /etc/prometheus/obs-zone-universe-main-dev/targets/postgres_exporter.yml
[OK]   ✓ postgres_exporter registered as Prometheus scrape target

[INFO] 3️⃣ Setting up pgbouncer_exporter integration...
[INFO] PgBouncer FQDN found in /etc/hosts: db-zone-universe-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.78
[INFO] PgBouncer detected: db-zone-universe-main-dev-postgresql-bouncer.fastorder.com:6432
[OK]   ✓ pgbouncer_exporter already installed
[INFO] Getting pgbouncer_admin password (SERVICE=zone, ZONE=universe)
[OK]   ✓ pgbouncer_admin password retrieved (24 chars)
[INFO] Using pgbouncer certs from: /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer
[INFO] Creating pgbouncer_exporter systemd service...
[OK]   ✓ pgbouncer_exporter service file created
[INFO] Starting pgbouncer_exporter service...
[2026-02-05 10:50:52 UTC] USER=www-data EUID=0 PID=888287 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 10:50:53 UTC] USER=www-data EUID=0 PID=888359 ACTION=passthru ARGS=systemctl enable pgbouncer_exporter-zone-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/pgbouncer_exporter-zone-universe-main-dev.service → /etc/systemd/system/pgbouncer_exporter-zone-universe-main-dev.service.
[2026-02-05 10:50:53 UTC] USER=www-data EUID=0 PID=888435 ACTION=passthru ARGS=systemctl restart pgbouncer_exporter-zone-universe-main-dev.service
[OK]   ✓ pgbouncer_exporter service running
[INFO] Registering pgbouncer_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: pgbouncer_exporter -> 10.100.1.78:9127
[OK]   ✓ Registered pgbouncer_exporter scrape target: 10.100.1.78:9127
[INFO]   Target file: /etc/prometheus/obs-zone-universe-main-dev/targets/pgbouncer_exporter.yml
[OK]   ✓ pgbouncer_exporter registered as Prometheus scrape target

[INFO] 4️⃣ Registering nodes to monitoring database...
[INFO] PostgreSQL key permissions set for www-data access: /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/postgres.key
[INFO] Registering PostgreSQL coordinator to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        zone-universe-main-dev-postgresql-coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.55
[INFO]   Port:              5432
[INFO]   FQDN:              db-zone-universe-main-dev-postgresql-coordinator.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 21addfb9-6a44-4dea-8573-7e2aa80db268
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   ✓ PostgreSQL coordinator registered
[INFO] Registering PgBouncer to monitoring dashboard...
[INFO]   FQDN: db-zone-universe-main-dev-postgresql-bouncer.fastorder.com, IP: 10.100.1.78, Port: 6432
[INFO]   Key permissions set for www-data access
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        zone-universe-main-dev-pgbouncer
[INFO]   Identifier Parent: pooling
[INFO]   IP:                10.100.1.78
[INFO]   Port:              6432
[INFO]   FQDN:              db-zone-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 80d2b3fc-6c2e-432b-ab5b-883cbc58f827
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   ✓ PgBouncer registered

[INFO] 5️⃣ Creating PgBouncer professional monitoring rules...
[INFO] Creating PgBouncer recording rules...
[OK]   ✓ PgBouncer recording rules created
[INFO] Creating PgBouncer alert rules with runbook URLs...
[OK]   ✓ PgBouncer alert rules with runbook URLs created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] 6️⃣ Setting up pgbackrest_exporter integration...
[INFO] pgBackRest detected, setting up exporter...
[OK]   ✓ pgbackrest_exporter already installed
[INFO] Creating pgbackrest_exporter systemd service...
[OK]   ✓ pgbackrest_exporter service file created
[INFO] Starting pgbackrest_exporter service...
[2026-02-05 10:50:58 UTC] USER=www-data EUID=0 PID=888756 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 10:50:59 UTC] USER=www-data EUID=0 PID=888806 ACTION=passthru ARGS=systemctl enable pgbackrest_exporter-zone-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/pgbackrest_exporter-zone-universe-main-dev.service → /etc/systemd/system/pgbackrest_exporter-zone-universe-main-dev.service.
[2026-02-05 10:50:59 UTC] USER=www-data EUID=0 PID=888857 ACTION=passthru ARGS=systemctl restart pgbackrest_exporter-zone-universe-main-dev.service
[WARN] ⚠️  pgbackrest_exporter service not running (may need manual start)
[WARN]     Run: systemctl status pgbackrest_exporter-zone-universe-main-dev.service
[INFO] Creating pgBackRest alert rules...
[OK]   ✓ pgBackRest alert rules created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ PostgreSQL & PgBouncer Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] postgres_exporter: http://localhost:9187/metrics
[INFO] pgbouncer_exporter: http://localhost:9127/metrics
[INFO] Prometheus: https://metrics-zone-universe-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-zone-universe-main-dev.fastorder.com
[INFO] 
[INFO] PgBouncer Monitoring:
[INFO]   • Recording rules: /etc/prometheus/obs-zone-universe-main-dev/rules/pgbouncer_recording_rules.yml
[INFO]   • Alert rules: /etc/prometheus/obs-zone-universe-main-dev/rules/pgbouncer_alerts.yml
[INFO] 
[INFO] pgBackRest Monitoring:
[INFO]   • Alert rules: /etc/prometheus/obs-zone-universe-main-dev/rules/pgbackrest_alerts.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up monitoring for 1 worker(s) and 1 standby(s) per worker...
[INFO] Setting up monitoring for: worker-01
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 PostgreSQL Monitoring Integration for zone-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-zone-universe-main-dev
[OK]   Observability cell endpoints registered for zone-universe-main-dev
[OK]   ✓ Observability cell is ready

[INFO] ✓ Using private IP for metrics: 10.100.1.55
[INFO] 2️⃣ Setting up postgres_exporter integration...
[INFO] Checking observability cell readiness: obs-zone-universe-main-dev
[OK]   Observability cell endpoints registered for zone-universe-main-dev
[INFO] Setting up postgres_exporter for zone-universe-main-dev
[2026-02-05 10:51:04 UTC] USER=www-data EUID=0 PID=889186 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-zone-universe-main-dev.yaml /etc/prometheus/postgres_exporter_queries-zone-universe-main-dev.yaml
[2026-02-05 10:51:04 UTC] USER=www-data EUID=0 PID=889199 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-zone-universe-main-dev.yaml
[2026-02-05 10:51:04 UTC] USER=www-data EUID=0 PID=889209 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-zone-universe-main-dev.yaml
[OK]   Custom queries file created at /etc/prometheus/postgres_exporter_queries-zone-universe-main-dev.yaml
[OK]   postgres_exporter already running with custom queries for zone-universe-main-dev
[OK]   ✓ postgres_exporter integration complete
[INFO] Registering postgres_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: postgres_exporter -> 10.100.1.55:9187
[OK]   ✓ Registered postgres_exporter scrape target: 10.100.1.55:9187
[INFO]   Target file: /etc/prometheus/obs-zone-universe-main-dev/targets/postgres_exporter.yml
[OK]   ✓ postgres_exporter registered as Prometheus scrape target

[INFO] 3️⃣ Setting up pgbouncer_exporter integration...
[INFO] PgBouncer FQDN found in /etc/hosts: db-zone-universe-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.78
[INFO] PgBouncer detected: db-zone-universe-main-dev-postgresql-bouncer.fastorder.com:6432
[OK]   ✓ pgbouncer_exporter already installed
[INFO] Getting pgbouncer_admin password (SERVICE=zone, ZONE=universe)
[OK]   ✓ pgbouncer_admin password retrieved (24 chars)
[INFO] Using pgbouncer certs from: /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer
[INFO] Creating pgbouncer_exporter systemd service...
[OK]   ✓ pgbouncer_exporter service file created
[INFO] Starting pgbouncer_exporter service...
[2026-02-05 10:51:06 UTC] USER=www-data EUID=0 PID=889359 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 10:51:07 UTC] USER=www-data EUID=0 PID=889420 ACTION=passthru ARGS=systemctl enable pgbouncer_exporter-zone-universe-main-dev.service
[2026-02-05 10:51:07 UTC] USER=www-data EUID=0 PID=889476 ACTION=passthru ARGS=systemctl restart pgbouncer_exporter-zone-universe-main-dev.service
[OK]   ✓ pgbouncer_exporter service running
[INFO] Registering pgbouncer_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: pgbouncer_exporter -> 10.100.1.78:9127
[OK]   ✓ Registered pgbouncer_exporter scrape target: 10.100.1.78:9127
[INFO]   Target file: /etc/prometheus/obs-zone-universe-main-dev/targets/pgbouncer_exporter.yml
[OK]   ✓ pgbouncer_exporter registered as Prometheus scrape target

[INFO] 4️⃣ Registering nodes to monitoring database...
[INFO] PostgreSQL key permissions set for www-data access: /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/postgres.key
[INFO] Registering PostgreSQL worker-01 to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        zone-universe-main-dev-postgresql-worker-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.55
[INFO]   Port:              5432
[INFO]   FQDN:              db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: a3189020-6bec-4e6c-9249-7f6d6b4de0f1
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   ✓ PostgreSQL worker-01 registered
[INFO] Registering PgBouncer to monitoring dashboard...
[INFO]   FQDN: db-zone-universe-main-dev-postgresql-bouncer.fastorder.com, IP: 10.100.1.78, Port: 6432
[INFO]   Key permissions set for www-data access
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        zone-universe-main-dev-pgbouncer
[INFO]   Identifier Parent: pooling
[INFO]   IP:                10.100.1.78
[INFO]   Port:              6432
[INFO]   FQDN:              db-zone-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 80d2b3fc-6c2e-432b-ab5b-883cbc58f827
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   ✓ PgBouncer registered

[INFO] 5️⃣ Creating PgBouncer professional monitoring rules...
[INFO] Creating PgBouncer recording rules...
[OK]   ✓ PgBouncer recording rules created
[INFO] Creating PgBouncer alert rules with runbook URLs...
[OK]   ✓ PgBouncer alert rules with runbook URLs created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] 6️⃣ Setting up pgbackrest_exporter integration...
[INFO] pgBackRest detected, setting up exporter...
[OK]   ✓ pgbackrest_exporter already installed
[INFO] Creating pgbackrest_exporter systemd service...
[OK]   ✓ pgbackrest_exporter service file created
[INFO] Starting pgbackrest_exporter service...
[2026-02-05 10:51:12 UTC] USER=www-data EUID=0 PID=889714 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 10:51:12 UTC] USER=www-data EUID=0 PID=889762 ACTION=passthru ARGS=systemctl enable pgbackrest_exporter-zone-universe-main-dev.service
[2026-02-05 10:51:13 UTC] USER=www-data EUID=0 PID=889816 ACTION=passthru ARGS=systemctl restart pgbackrest_exporter-zone-universe-main-dev.service
[WARN] ⚠️  pgbackrest_exporter service not running (may need manual start)
[WARN]     Run: systemctl status pgbackrest_exporter-zone-universe-main-dev.service
[INFO] Creating pgBackRest alert rules...
[OK]   ✓ pgBackRest alert rules created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ PostgreSQL & PgBouncer Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] postgres_exporter: http://localhost:9187/metrics
[INFO] pgbouncer_exporter: http://localhost:9127/metrics
[INFO] Prometheus: https://metrics-zone-universe-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-zone-universe-main-dev.fastorder.com
[INFO] 
[INFO] PgBouncer Monitoring:
[INFO]   • Recording rules: /etc/prometheus/obs-zone-universe-main-dev/rules/pgbouncer_recording_rules.yml
[INFO]   • Alert rules: /etc/prometheus/obs-zone-universe-main-dev/rules/pgbouncer_alerts.yml
[INFO] 
[INFO] pgBackRest Monitoring:
[INFO]   • Alert rules: /etc/prometheus/obs-zone-universe-main-dev/rules/pgbackrest_alerts.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up monitoring for standby: worker-01-standby-01
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 PostgreSQL Monitoring Integration for zone-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-zone-universe-main-dev
[OK]   Observability cell endpoints registered for zone-universe-main-dev
[OK]   ✓ Observability cell is ready

[INFO] ✓ Using private IP for metrics: 10.100.1.55
[INFO] 2️⃣ Setting up postgres_exporter integration...
[INFO] Checking observability cell readiness: obs-zone-universe-main-dev
[OK]   Observability cell endpoints registered for zone-universe-main-dev
[INFO] Setting up postgres_exporter for zone-universe-main-dev
[2026-02-05 10:51:17 UTC] USER=www-data EUID=0 PID=890173 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-zone-universe-main-dev.yaml /etc/prometheus/postgres_exporter_queries-zone-universe-main-dev.yaml
[2026-02-05 10:51:17 UTC] USER=www-data EUID=0 PID=890184 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-zone-universe-main-dev.yaml
[2026-02-05 10:51:17 UTC] USER=www-data EUID=0 PID=890199 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-zone-universe-main-dev.yaml
[OK]   Custom queries file created at /etc/prometheus/postgres_exporter_queries-zone-universe-main-dev.yaml
[OK]   postgres_exporter already running with custom queries for zone-universe-main-dev
[OK]   ✓ postgres_exporter integration complete
[INFO] Registering postgres_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: postgres_exporter -> 10.100.1.55:9187
[OK]   ✓ Registered postgres_exporter scrape target: 10.100.1.55:9187
[INFO]   Target file: /etc/prometheus/obs-zone-universe-main-dev/targets/postgres_exporter.yml
[OK]   ✓ postgres_exporter registered as Prometheus scrape target

[INFO] 3️⃣ Setting up pgbouncer_exporter integration...
[INFO] PgBouncer FQDN found in /etc/hosts: db-zone-universe-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.78
[INFO] PgBouncer detected: db-zone-universe-main-dev-postgresql-bouncer.fastorder.com:6432
[OK]   ✓ pgbouncer_exporter already installed
[INFO] Getting pgbouncer_admin password (SERVICE=zone, ZONE=universe)
[OK]   ✓ pgbouncer_admin password retrieved (24 chars)
[INFO] Using pgbouncer certs from: /etc/fastorder/postgresql/certs/zone-universe-main-dev/pgbouncer
[INFO] Creating pgbouncer_exporter systemd service...
[OK]   ✓ pgbouncer_exporter service file created
[INFO] Starting pgbouncer_exporter service...
[2026-02-05 10:51:20 UTC] USER=www-data EUID=0 PID=890366 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 10:51:20 UTC] USER=www-data EUID=0 PID=890425 ACTION=passthru ARGS=systemctl enable pgbouncer_exporter-zone-universe-main-dev.service
[2026-02-05 10:51:21 UTC] USER=www-data EUID=0 PID=890482 ACTION=passthru ARGS=systemctl restart pgbouncer_exporter-zone-universe-main-dev.service
[OK]   ✓ pgbouncer_exporter service running
[INFO] Registering pgbouncer_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: pgbouncer_exporter -> 10.100.1.78:9127
[OK]   ✓ Registered pgbouncer_exporter scrape target: 10.100.1.78:9127
[INFO]   Target file: /etc/prometheus/obs-zone-universe-main-dev/targets/pgbouncer_exporter.yml
[OK]   ✓ pgbouncer_exporter registered as Prometheus scrape target

[INFO] 4️⃣ Registering nodes to monitoring database...
[INFO] Registering PostgreSQL worker-01-standby-01 to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        zone-universe-main-dev-postgresql-worker-01-standby-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.55
[INFO]   Port:              5432
[INFO]   FQDN:              db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 9141c678-a9ea-4c62-992a-6c1b13f6271c
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   ✓ PostgreSQL worker-01-standby-01 registered
[INFO] Registering PgBouncer to monitoring dashboard...
[INFO]   FQDN: db-zone-universe-main-dev-postgresql-bouncer.fastorder.com, IP: 10.100.1.78, Port: 6432
[INFO]   Key permissions set for www-data access
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        zone-universe-main-dev-pgbouncer
[INFO]   Identifier Parent: pooling
[INFO]   IP:                10.100.1.78
[INFO]   Port:              6432
[INFO]   FQDN:              db-zone-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 80d2b3fc-6c2e-432b-ab5b-883cbc58f827
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   ✓ PgBouncer registered

[INFO] 5️⃣ Creating PgBouncer professional monitoring rules...
[INFO] Creating PgBouncer recording rules...
[OK]   ✓ PgBouncer recording rules created
[INFO] Creating PgBouncer alert rules with runbook URLs...
[OK]   ✓ PgBouncer alert rules with runbook URLs created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] 6️⃣ Setting up pgbackrest_exporter integration...
[INFO] pgBackRest detected, setting up exporter...
[OK]   ✓ pgbackrest_exporter already installed
[INFO] Creating pgbackrest_exporter systemd service...
[OK]   ✓ pgbackrest_exporter service file created
[INFO] Starting pgbackrest_exporter service...
[2026-02-05 10:51:26 UTC] USER=www-data EUID=0 PID=890799 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 10:51:26 UTC] USER=www-data EUID=0 PID=890850 ACTION=passthru ARGS=systemctl enable pgbackrest_exporter-zone-universe-main-dev.service
[2026-02-05 10:51:27 UTC] USER=www-data EUID=0 PID=890911 ACTION=passthru ARGS=systemctl restart pgbackrest_exporter-zone-universe-main-dev.service
[WARN] ⚠️  pgbackrest_exporter service not running (may need manual start)
[WARN]     Run: systemctl status pgbackrest_exporter-zone-universe-main-dev.service
[INFO] Creating pgBackRest alert rules...
[OK]   ✓ pgBackRest alert rules created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ PostgreSQL & PgBouncer Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] postgres_exporter: http://localhost:9187/metrics
[INFO] pgbouncer_exporter: http://localhost:9127/metrics
[INFO] Prometheus: https://metrics-zone-universe-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-zone-universe-main-dev.fastorder.com
[INFO] 
[INFO] PgBouncer Monitoring:
[INFO]   • Recording rules: /etc/prometheus/obs-zone-universe-main-dev/rules/pgbouncer_recording_rules.yml
[INFO]   • Alert rules: /etc/prometheus/obs-zone-universe-main-dev/rules/pgbouncer_alerts.yml
[INFO] 
[INFO] pgBackRest Monitoring:
[INFO]   • Alert rules: /etc/prometheus/obs-zone-universe-main-dev/rules/pgbackrest_alerts.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ Monitoring setup completed for coordinator, workers, and standbys

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 12-setup-offsite-backup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Setting up offsite backup repository for zone-universe-main-dev...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Offsite Backup Repository Setup (repo2)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📋 OFFSITE BACKUP INFORMATION
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Why Offsite Backups?
[INFO]   ✓ Disaster recovery resilience (datacenter loss, hardware failure)
[INFO]   ✓ Protection against local corruption or ransomware
[INFO]   ✓ Compliance requirements (geographic redundancy)
[INFO]   ✓ Long-term archival with cost-effective storage tiers

[WARN] ⚠️  Offsite backup (repo2) is NOT ENABLED
[WARN]    Using local backups only (repo1)

[INFO] Configuration Example Location:
[INFO]   📄 /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example

[INFO] Supported Storage Backends:
[INFO]   • AWS S3 (standard, multi-region)
[INFO]   • AWS S3 Glacier (low-cost archival)
[INFO]   • MinIO (self-hosted S3-compatible)
[INFO]   • Google Cloud Storage (via S3 compatibility)
[INFO]   • Azure Blob Storage (via S3 compatibility)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📝 SETUP INSTRUCTIONS
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Step 1: Review the example configuration
[INFO]   cat /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example

[INFO] Step 2: Prepare S3 bucket and credentials
[INFO]   • Create S3 bucket (or MinIO bucket)
[INFO]   • Create IAM user with S3 permissions (PutObject, GetObject, DeleteObject, ListBucket)
[INFO]   • Note: Access Key ID and Secret Access Key

[INFO] Step 3: Add repo2 configuration to /etc/pgbackrest/pgbackrest.conf
[INFO]   • Copy repo2-* settings from example to [global] section
[INFO]   • Replace placeholders (bucket name, access keys, region)
[INFO]   • Note: Use same cipher key as repo1, or generate separate key for repo2

[INFO] Step 4: Initialize repo2 stanzas
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=zone-universe-main-dev-coordinator stanza-create --repo=2
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=zone-universe-main-dev-worker-01 stanza-create --repo=2
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=zone-universe-main-dev-worker-02 stanza-create --repo=2

[INFO] Step 5: Verify repo2 configuration
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=zone-universe-main-dev-coordinator check --repo=2

[INFO] Step 6: Take initial full backup to repo2
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=zone-universe-main-dev-coordinator --repo=2 --type=full backup

[INFO] Step 7: Update backup automation to include repo2
[INFO]   • Edit: /usr/local/bin/pgbackrest-full-backup-zone-universe-main-dev.sh
[INFO]   • Change: pgbackrest backup to pgbackrest --repo=1,2 backup
[INFO]   • Or: Add separate cron for repo2 backups

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🧪 TESTING
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] After configuration, run:
[INFO]   ./08-setup-offsite-backup.sh test

[INFO] This will verify:
[INFO]   ✓ S3 connectivity
[INFO]   ✓ Stanza initialization
[INFO]   ✓ Test backup and restore from repo2

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 💡 COST OPTIMIZATION
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] AWS S3 Lifecycle Policies (transition to cheaper storage):
[INFO]   • 0-30 days:   S3 Standard (~$0.023/GB/month)
[INFO]   • 30-90 days:  S3 Standard-IA (~$0.0125/GB/month)
[INFO]   • 90+ days:    S3 Glacier (~$0.004/GB/month)

[INFO] Estimated costs for 100GB backups:
[INFO]   • All Standard:     ~$2.30/month
[INFO]   • With lifecycle:   ~$1.20/month (48% savings)


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 13-setup-monitoring-alerts.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Setting up backup monitoring and alerting for zone-universe-main-dev...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Monitoring and Alerting Configuration
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] This will set up monitoring for:
  • Backup failures (cron job failures)
  • WAL archiving backlog (>100 files)
  • Repository disk space (<20% free)
  • Backup age (>25 hours)

[INFO] No alert email configured (set ALERT_EMAIL environment variable)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Creating monitoring directories...
[2026-02-05 10:51:35 UTC] USER=www-data EUID=0 PID=891299 ACTION=fsop ARGS=mkdir -p /opt/pgbackrest-monitoring
[2026-02-05 10:51:35 UTC] USER=www-data EUID=0 PID=891351 ACTION=fsop ARGS=chmod 777 /var/log/pgbackrest-monitoring
[2026-02-05 10:51:35 UTC] USER=www-data EUID=0 PID=891366 ACTION=fsop ARGS=chown postgres:postgres /opt/pgbackrest-monitoring
[2026-02-05 10:51:35 UTC] USER=www-data EUID=0 PID=891400 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest-monitoring
[INFO] ✅ Directories created

[INFO] 2️⃣ Creating alert helper script...
[2026-02-05 10:51:35 UTC] USER=www-data EUID=0 PID=891467 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/send-alert.sh
[INFO] ✅ Alert helper created

[INFO] 3️⃣ Creating WAL queue monitoring script...
[2026-02-05 10:51:35 UTC] USER=www-data EUID=0 PID=891539 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-wal-queue.sh
[INFO] ✅ WAL queue monitor created

[INFO] 4️⃣ Creating backup age monitoring script...
[2026-02-05 10:51:35 UTC] USER=www-data EUID=0 PID=891563 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-backup-age.sh
[INFO] ✅ Backup age monitor created

[INFO] 5️⃣ Creating repository disk space monitoring script...
[2026-02-05 10:51:36 UTC] USER=www-data EUID=0 PID=891583 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-repo-space.sh
[INFO] ✅ Disk space monitor created

[INFO] 6️⃣ Creating backup failure detection script...
[2026-02-05 10:51:36 UTC] USER=www-data EUID=0 PID=891605 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-backup-failures.sh
[INFO] ✅ Backup failure detector created

[INFO] 7️⃣ Creating master monitoring script...
[2026-02-05 10:51:36 UTC] USER=www-data EUID=0 PID=891623 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO] ✅ Master monitoring script created

[INFO] 8️⃣ Installing mailutils for email alerts...
[INFO] ✅ mailutils already installed

[INFO] 9️⃣ Installing jq for JSON parsing...
[INFO] ✅ jq already installed

[INFO] 🔟 Setting up monitoring cron jobs...
[INFO] ✅ Monitoring cron jobs configured
[INFO]    Checks run every 15 minutes

[INFO] 1️⃣1️⃣ Creating monitoring dashboard...
[2026-02-05 10:51:36 UTC] USER=www-data EUID=0 PID=891662 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/dashboard.sh
[INFO] ✅ Monitoring dashboard created

[INFO] 1️⃣2️⃣ Running initial monitoring check...

[2026-02-05 10:51:36 UTC] USER=www-data EUID=0 PID=891671 ACTION=passthru ARGS=bash /opt/pgbackrest-monitoring/run-all-checks.sh

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup monitoring setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Monitoring Configuration:
[INFO]   Alert Email:        
[INFO]   Slack Webhook:      Not configured

[INFO] Monitoring Checks:
[INFO]   • WAL Queue:        Every 15 minutes (threshold: >100 files)
[INFO]   • Backup Age:       Every 15 minutes (threshold: >25 hours)
[INFO]   • Disk Space:       Every 15 minutes (threshold: <20% free)
[INFO]   • Backup Failures:  Every 15 minutes (log analysis)

[INFO] Scripts Created:
[INFO]   Monitoring dir:     /opt/pgbackrest-monitoring
[INFO]   Log dir:            /var/log/pgbackrest-monitoring
[INFO]   Dashboard:          /opt/pgbackrest-monitoring/dashboard.sh
[INFO]   Master check:       /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO]   Alert sender:       /opt/pgbackrest-monitoring/send-alert.sh

[INFO] Useful Commands:
[INFO]   View dashboard:     /usr/local/bin/fastorder-provisioning-wrapper.sh /opt/pgbackrest-monitoring/dashboard.sh
[INFO]   Run checks now:     /usr/local/bin/fastorder-provisioning-wrapper.sh /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO]   View alerts:        tail -f /var/log/pgbackrest-monitoring/alerts.log
[INFO]   View monitoring:    tail -f /var/log/pgbackrest-monitoring/monitoring.log

[INFO] Cron Schedule:
[INFO]   All checks:         Every 15 minutes
[INFO]   Log rotation:       Weekly (keep 7 days)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 14-vault-cipher-key.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] ✅ Using permanent AWS credentials from /home/ab/.aws/credentials [default] profile
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔐 PostgreSQL Cipher Key Vaulting
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Environment:       zone-universe-main-dev
[INFO]   AWS Region:        me-central-1
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣  Verifying AWS setup...
[INFO] ✅ AWS authentication successful

[INFO] 2️⃣  Verifying cipher key...
[INFO] ✅ Cipher key found
[INFO]    Location: /etc/pgbackrest/.cipher-key-zone-universe-main-dev
[INFO]    Hash (MD5): 66e1cecd0273db783a054aecb9560779
[INFO]    Size: 150 bytes

[INFO] 3️⃣  Vaulting cipher key to AWS Secrets Manager...
[INFO]    Secret name: fastorder/db/zone/universe/main/dev/postgresql/pgbackrest/cipher-key
[INFO]    Creating new secret...
[INFO] ✅ Cipher key stored in AWS Secrets Manager
[INFO]    Verifying storage...
[INFO] ✅ Verification successful - key matches

[INFO] 4️⃣  Creating local encrypted backup...
[2026-02-05 10:51:49 UTC] USER=www-data EUID=0 PID=892267 ACTION=fsop ARGS=mv /tmp/cipher-key-backup-891936.enc /root/.pgbackrest-cipher-key-zone-universe-main-dev.enc
[2026-02-05 10:51:49 UTC] USER=www-data EUID=0 PID=892276 ACTION=fsop ARGS=chmod 600 /root/.pgbackrest-cipher-key-zone-universe-main-dev.enc
[INFO] ✅ Local encrypted backup created
[INFO]    Backup file: /root/.pgbackrest-cipher-key-zone-universe-main-dev.enc
[INFO]    Passphrase: /root/.pgbackrest-cipher-key-passphrase-zone-universe-main-dev.txt

[INFO] 5️⃣  Vaulting backup passphrase...
[INFO] ✅ Backup passphrase vaulted

[INFO] 6️⃣  Creating recovery documentation...
[2026-02-05 10:51:53 UTC] USER=www-data EUID=0 PID=892406 ACTION=fsop ARGS=chmod 640 /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_zone-universe-main-dev.md
[2026-02-05 10:51:53 UTC] USER=www-data EUID=0 PID=892417 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_zone-universe-main-dev.md
[INFO] ✅ Recovery documentation: /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_zone-universe-main-dev.md

[INFO] 7️⃣  Storing backup metadata...
[INFO] ✅ Backup metadata stored in AWS Secrets Manager
[INFO]    Secret: fastorder/db/zone/universe/main/dev/postgresql/backup/metadata-20260205-105153

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Cipher Key Vaulting Complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO]   Environment:          zone-universe-main-dev
[INFO]   Key Hash:             66e1cecd0273db783a054aecb9560779

[INFO] AWS Secrets:
[INFO]   Cipher Key:           fastorder/db/zone/universe/main/dev/postgresql/pgbackrest/cipher-key
[INFO]   Passphrase:           fastorder/db/zone/universe/main/dev/postgresql/pgbackrest/cipher-key-passphrase
[INFO]   Backup Metadata:      fastorder/db/zone/universe/main/dev/postgresql/backup/metadata-20260205-105153

[INFO] Local Backups:
[INFO]   Encrypted File:       /root/.pgbackrest-cipher-key-zone-universe-main-dev.enc
[INFO]   Passphrase File:      /root/.pgbackrest-cipher-key-passphrase-zone-universe-main-dev.txt

[INFO] Recovery Doc:           /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_zone-universe-main-dev.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Skipping 15-backup-restore-test.sh (test script - set RUN_TESTS=true to enable)
[INFO] Skipping 16-test-recovery.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 17-verification.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)

[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] PostgreSQL Production Readiness Verification
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] 
[INFO] This script verifies 3 CRITICAL checks for production readiness:
[INFO]   1. Citus Cluster Operational (coordinator + workers)
[INFO]   2. SSL/TLS Enforced (certificates valid, connections secure)
[INFO]   3. Coordinator Backups Configured (pgBackRest functional)
[INFO] 
[INFO] 📖 Documentation: /tmp/VERIFICATION_RUNBOOK.md
[INFO] 🔐 Security: Uses sudo for certificate checks (maintains strict permissions)
[INFO] 📊 Exit Code: 0 = production ready, 1 = critical checks failed
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🕐 Ensuring all PostgreSQL services are ready...
[ OK ] ✅ All PostgreSQL services are ready

[INFO] 🔍 Starting PostgreSQL verification...
[INFO] Environment: zone-universe-main-dev
[INFO] Citus: yes

[INFO] Citus mode ENABLED
[INFO] → Coordinator + 1 worker(s) + 3 HA node(s) per worker

[INFO] Verifying 1 worker(s)...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying: worker-01 (type: worker-01)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Starting PostgreSQL verification for zone-universe-main-dev-worker-01...

[INFO] 1️⃣ Checking systemd service status...
[OK]   ✅ Service postgresql@zone-universe-main-dev-worker-01.service is active

[INFO] 2️⃣ Checking PostgreSQL process...
[OK]   ✅ PostgreSQL process is running

[INFO] 3️⃣ Checking socket directory...
[OK]   ✅ Socket directory exists: /var/run/postgresql-zone-universe-main-dev-worker-01
total 4
drwxrwsr-x  2 postgres postgres   80 Feb  5 10:44 .
drwxr-xr-x 67 root     root     1740 Feb  5 10:51 ..
srwxrwxrwx  1 postgres postgres    0 Feb  5 10:44 .s.PGSQL.5432
-rw-------  1 postgres postgres  129 Feb  5 10:44 .s.PGSQL.5432.lock

[INFO] 4️⃣ Testing connection via Unix socket...
[OK]   ✅ Socket connection successful
                                                              version                                                              
-----------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.6 (Ubuntu 17.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

[INFO] 5️⃣ Checking SSL certificates...
[2026-02-05 10:51:59 UTC] USER=www-data EUID=0 PID=892841 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.crt
[OK]   ✅ Server certificate exists: /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.crt
[2026-02-05 10:51:59 UTC] USER=www-data EUID=0 PID=892851 ACTION=fsop ARGS=openssl x509 -in /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/server.crt -noout -checkend 86400
[OK]   ✅ Server certificate is valid
[2026-02-05 10:51:59 UTC] USER=www-data EUID=0 PID=892860 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt
[OK]   ✅ CA certificate exists: /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01/ca.crt
[INFO] ℹ️  Client certificates not found at /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/postgres.crt
[INFO]     (This is OK if using password authentication)

[INFO] 6️⃣ Checking PostgreSQL settings...
[OK]   ✅ SSL is enabled worker-01 worker-01
[OK]   ✅ Max connections: 100
[OK]   ✅ Listen addresses: 10.100.1.106
[OK]   ✅ WAL level: logical
[OK]   ✅ Shared preload libraries: shared_preload_libraries

[INFO] 7️⃣ Checking replication configuration...
[INFO] ℹ️  No synchronous standbys configured (single node or async replication)
[INFO] Checking replication slots...
      slot_name       | slot_type | active | restart_lsn 
----------------------+-----------+--------+-------------
 worker_01_standby_01 | physical  | f      | 
(1 row)
[OK]   ✅ Replication slot naming uses underscores (correct)
[INFO] Checking active replication connections...
 application_name | client_addr | state | sync_state 
------------------+-------------+-------+------------
(0 rows)
[INFO] ℹ️  No active replication connections
[INFO] ℹ️  This is a PRIMARY node (no standby.signal)

[INFO] 8️⃣ Checking pg_hba.conf for replication rules...
[WARN] ⚠️ pg_hba.conf not found at /data/postgresql/17/zone-universe-main-dev/worker-01/pg_hba.conf

[INFO] 9️⃣ Checking Citus configuration...
[OK]   ✅ Citus extension is installed
[OK]   ✅ Citus version: Citus 13.2.0
[OK]   ✅ max_prepared_transactions: 100 (adequate for Citus)
[INFO] Citus active worker nodes:
                          node_name                           | node_port 
--------------------------------------------------------------+-----------
 db-zone-universe-main-dev-postgresql-worker-01.fastorder.com |      5432
(1 row)



[INFO] 🔟 Checking data directory...
[OK]   ✅ Data directory exists: /data/postgresql/17/zone-universe-main-dev/worker-01
[OK]   ✅ Data directory size: 4.0K

[INFO] 1️⃣1️⃣ Checking PgBouncer configuration...
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/zone-universe-main-dev/pgbouncer.ini' as root on web-03.
Failed to print table: Broken pipe
[OK]   ✅ PgBouncer is installed
[INFO]    Version: 1.24.1
2.1.12-stable
c-ares
OpenSSL
yes
[OK]   ✅ PgBouncer service is active: pgbouncer@zone-universe-main-dev.service
[WARN] ⚠️ PgBouncer IP service is not active: pgbouncer-ip@zone-universe-main-dev.service
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/zone-universe-main-dev/pgbouncer.ini' as root on web-03.
[WARN] ⚠️ PgBouncer config not found: /etc/pgbouncer/zone-universe-main-dev/pgbouncer.ini
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/zone-universe-main-dev/userlist.txt' as root on web-03.
[WARN] ⚠️ PgBouncer auth file not found: /etc/pgbouncer/zone-universe-main-dev/userlist.txt
[OK]   ✅ PgBouncer is listening on port 6432

[INFO] 1️⃣2️⃣ Enhanced PgBouncer Admin Console Verification...
Failed to print table: Broken pipe
[INFO] ℹ️  PgBouncer not configured for enhanced verification

[INFO] 1️⃣3️⃣ Replicator User Connection Verification...
[INFO] Found 1 replication slot(s) - verifying replicator connectivity...
[WARN] ⚠️ Replicator certificates not found at /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01
[INFO]    Expected files:
[INFO]    - /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/root.crt
[INFO]    - /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.crt
[INFO]    - /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01/replicator.key
[INFO] Checking pg_hba.conf replicator rules...
[OK]   ✅ Replicator HBA rules found:
 line_number |  type   |   database    |  user_name   |   address    |  auth_method  | options | error 
-------------+---------+---------------+--------------+--------------+---------------+---------+-------
          20 | hostssl | {replication} | {replicator} | 10.100.1.196 | scram-sha-256 |         | 
          21 | hostssl | {replication} | {replicator} | 10.100.1.106 | scram-sha-256 |         | 
(2 rows)
[INFO] Checking active replicator connections in pg_stat_activity...
[WARN] ⚠️ No active replicator connections in pg_stat_activity
[WARN]    This is expected if standbys are not currently connected

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ PostgreSQL verification completed successfully!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Instance:       zone-universe-main-dev-worker-01
[INFO] Service:        postgresql@zone-universe-main-dev-worker-01.service
[INFO] Socket:         /var/run/postgresql-zone-universe-main-dev-worker-01
[INFO] Data Directory: /data/postgresql/17/zone-universe-main-dev/worker-01
[INFO] Hostname:       db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
[INFO] Port:           5432
[INFO] SSL:            on
[INFO] WAL Level:      logical
[INFO] Citus:          yes
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 💡 OPTIMIZATION OPPORTUNITIES (Optional Enhancements)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1. Review connection limits for production workload
[INFO]    🔌 Current: max_connections = 100 (PostgreSQL default)
[INFO]    💡 Consider: Increasing to 200-500 for production applications
[INFO]    ⚙️  Alternative: Use PgBouncer connection pooling (lower PostgreSQL limit, higher client capacity)
[INFO]    🔧 Action: Adjust max_connections in postgresql.conf based on workload analysis
[INFO]    ⚠️  Note: Each connection consumes ~10MB RAM; tune based on available memory
[INFO]    📚 Docs: https://www.postgresql.org/docs/current/runtime-config-connection.html

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ℹ️  These are optional enhancements for production-scale deployments
[INFO] ℹ️  Current configuration is fully functional and ready for production
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] ✅ Verification passed for worker-01

Failed to print table: Broken pipe
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying: worker-01-standby-01 (type: worker-01)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Starting PostgreSQL verification for zone-universe-main-dev-worker-01-standby-01...

[INFO] 1️⃣ Checking systemd service status...
[OK]   ✅ Service postgresql@zone-universe-main-dev-worker-01-standby-01.service is active

[INFO] 2️⃣ Checking PostgreSQL process...
[OK]   ✅ PostgreSQL process is running

[INFO] 3️⃣ Checking socket directory...
[OK]   ✅ Socket directory exists: /var/run/postgresql-zone-universe-main-dev-worker-01-standby-01
total 4
drwxrwsr-x  2 postgres postgres   80 Feb  5 10:48 .
drwxr-xr-x 67 root     root     1740 Feb  5 10:52 ..
srwxrwxrwx  1 postgres postgres    0 Feb  5 10:48 .s.PGSQL.5432
-rw-------  1 postgres postgres  151 Feb  5 10:48 .s.PGSQL.5432.lock

[INFO] 4️⃣ Testing connection via Unix socket...
[OK]   ✅ Socket connection successful
                                                              version                                                              
-----------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.6 (Ubuntu 17.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

[INFO] 5️⃣ Checking SSL certificates...
[2026-02-05 10:52:19 UTC] USER=www-data EUID=0 PID=893924 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt
[OK]   ✅ Server certificate exists: /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt
[2026-02-05 10:52:19 UTC] USER=www-data EUID=0 PID=893940 ACTION=fsop ARGS=openssl x509 -in /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/server.crt -noout -checkend 86400
Certificate will not expire
[OK]   ✅ Server certificate is valid
[2026-02-05 10:52:19 UTC] USER=www-data EUID=0 PID=893951 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/ca.crt
[OK]   ✅ CA certificate exists: /etc/fastorder/postgresql/certs/zone-universe-main-dev/worker-01-standby-01/ca.crt
[INFO] ℹ️  Client certificates not found at /home/postgres/ssl/.postgresql/zone-universe-main-dev/worker-01-standby-01/postgres.crt
[INFO]     (This is OK if using password authentication)

[INFO] 6️⃣ Checking PostgreSQL settings...
[OK]   ✅ SSL is enabled worker-01 worker-01-standby-01
[OK]   ✅ Max connections: 100
[OK]   ✅ Listen addresses: 10.100.1.196
[OK]   ✅ WAL level: logical
[OK]   ✅ Shared preload libraries: shared_preload_libraries

[INFO] 7️⃣ Checking replication configuration...
[INFO] ℹ️  No synchronous standbys configured (single node or async replication)
[INFO] Checking replication slots...
psql: error: connection to server at "db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com" (10.100.1.196), port 5432 failed: SSL error: certificate verify failed
connection to server at "db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com" (10.100.1.196), port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "10.100.1.196", user "postgres", database "postgres", no encryption
[OK]   ✅ Replication slot naming uses underscores (correct)
[INFO] Checking active replication connections...
 application_name | client_addr | state | sync_state 
------------------+-------------+-------+------------
(0 rows)
[INFO] ℹ️  No active replication connections
[INFO] ℹ️  This is a PRIMARY node (no standby.signal)

[INFO] 8️⃣ Checking pg_hba.conf for replication rules...
[WARN] ⚠️ pg_hba.conf not found at /data/postgresql/17/zone-universe-main-dev/worker-01-standby-01/pg_hba.conf

[INFO] 9️⃣ Checking Citus configuration...
[INFO] ℹ️  Citus extension not needed on standby (will inherit from primary via replication)

[INFO] 🔟 Checking data directory...
[OK]   ✅ Data directory exists: /data/postgresql/17/zone-universe-main-dev/worker-01-standby-01
[OK]   ✅ Data directory size: 4.0K

[INFO] 1️⃣1️⃣ Checking PgBouncer configuration...
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/zone-universe-main-dev/pgbouncer.ini' as root on web-03.
Failed to print table: Broken pipe
[OK]   ✅ PgBouncer is installed
[INFO]    Version: 1.24.1
2.1.12-stable
c-ares
OpenSSL
yes
Failed to print table: Broken pipe
[INFO] ℹ️  PgBouncer service not configured for this environment

[INFO] 1️⃣2️⃣ Enhanced PgBouncer Admin Console Verification...
[INFO] ℹ️  PgBouncer password not found

[INFO] 1️⃣3️⃣ Replicator User Connection Verification...
[INFO] ℹ️  No replication slots configured - skipping replicator verification

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ PostgreSQL verification completed successfully!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Instance:       zone-universe-main-dev-worker-01-standby-01
[INFO] Service:        postgresql@zone-universe-main-dev-worker-01-standby-01.service
[INFO] Socket:         /var/run/postgresql-zone-universe-main-dev-worker-01-standby-01
[INFO] Data Directory: /data/postgresql/17/zone-universe-main-dev/worker-01-standby-01
[INFO] Hostname:       db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com
[INFO] Port:           5432
[INFO] SSL:            on
[INFO] WAL Level:      logical
[INFO] Citus:          yes
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] ✅ Verification passed for worker-01-standby-01

[INFO] Skipping worker-01-standby-02 - service not configured
[INFO] Skipping worker-01-standby-03 - service not configured
[INFO] Verifying coordinator...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying: coordinator (type: coordinator)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Starting PostgreSQL verification for zone-universe-main-dev-coordinator...

[INFO] 1️⃣ Checking systemd service status...
[OK]   ✅ Service postgresql@zone-universe-main-dev-coordinator.service is active

[INFO] 2️⃣ Checking PostgreSQL process...
[OK]   ✅ PostgreSQL process is running

[INFO] 3️⃣ Checking socket directory...
[OK]   ✅ Socket directory exists: /var/run/postgresql-zone-universe-main-dev-coordinator
total 4
drwxrwsr-x  2 postgres postgres   80 Feb  5 10:50 .
drwxr-xr-x 67 root     root     1740 Feb  5 10:52 ..
srwxrwxrwx  1 postgres postgres    0 Feb  5 10:50 .s.PGSQL.5432
-rw-------  1 postgres postgres  133 Feb  5 10:50 .s.PGSQL.5432.lock

[INFO] 4️⃣ Testing connection via Unix socket...
[OK]   ✅ Socket connection successful
                                                              version                                                              
-----------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.6 (Ubuntu 17.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

[INFO] 5️⃣ Checking SSL certificates...
[2026-02-05 10:52:37 UTC] USER=www-data EUID=0 PID=894897 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.crt
[OK]   ✅ Server certificate exists: /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/server.crt
[OK]   ✅ Server certificate is valid
[2026-02-05 10:52:37 UTC] USER=www-data EUID=0 PID=894921 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt
[OK]   ✅ CA certificate exists: /etc/fastorder/postgresql/certs/zone-universe-main-dev/coordinator/ca.crt
[INFO] ℹ️  Client certificates not found at /home/postgres/ssl/.postgresql/zone-universe-main-dev/coordinator/postgres.crt
[INFO]     (This is OK if using password authentication)

[INFO] 6️⃣ Checking PostgreSQL settings...
[OK]   ✅ SSL is enabled coordinator coordinator
[OK]   ✅ Max connections: 150
[OK]   ✅ Listen addresses: 10.100.1.55
[OK]   ✅ WAL level: logical
[OK]   ✅ Shared preload libraries: shared_preload_libraries

[INFO] 7️⃣ Checking replication configuration...
[INFO] ℹ️  No synchronous standbys configured (single node or async replication)
[INFO] Checking replication slots...
 slot_name | slot_type | active | restart_lsn 
-----------+-----------+--------+-------------
(0 rows)
[OK]   ✅ Replication slot naming uses underscores (correct)
[INFO] Checking active replication connections...
 application_name | client_addr | state | sync_state 
------------------+-------------+-------+------------
(0 rows)
[INFO] ℹ️  No active replication connections
[INFO] ℹ️  This is a PRIMARY node (no standby.signal)

[INFO] 8️⃣ Checking pg_hba.conf for replication rules...
[WARN] ⚠️ pg_hba.conf not found at /data/postgresql/17/zone-universe-main-dev/coordinator/pg_hba.conf

[INFO] 9️⃣ Checking Citus configuration...
[OK]   ✅ Citus extension is installed
[OK]   ✅ Citus version: Citus 13.2.0
[OK]   ✅ max_prepared_transactions: 100 (adequate for Citus)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] COORDINATOR-SPECIFIC CHECKS
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking registered workers...
[INFO] ℹ️  Coordinator role verified via pg_dist_node (1 workers registered)
[INFO] Checking coordinator hostname configuration...
[OK]   ✅ Coordinator hostname: ----------------------------------------------------------------:----------
[INFO] Checking for stuck prepared transactions...
[OK]   ✅ No stuck Citus prepared transactions
[INFO] Expected workers: 1
[INFO] Registered workers: 1
[OK]   ✅ All 1 worker(s) successfully registered
[INFO] Registered worker nodes:
psql: error: connection to server at "db-zone-universe-main-dev-postgresql.fastorder.com" (10.100.1.55), port 5432 failed: SSL error: certificate verify failed
connection to server at "db-zone-universe-main-dev-postgresql.fastorder.com" (10.100.1.55), port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "10.100.1.55", user "postgres", database "postgres", no encryption
[INFO] Note: groupid=0 is the coordinator, groupid>0 are workers

[INFO] Citus active worker nodes:
                          node_name                           | node_port 
--------------------------------------------------------------+-----------
 db-zone-universe-main-dev-postgresql-worker-01.fastorder.com |      5432
(1 row)


[INFO] Verifying Citus workers...
[INFO] Checking worker: db-zone-universe-main-dev-postgresql-worker-01.fastorder.com
 citus_add_node 
----------------
              2
(1 row)


[INFO] Testing Citus distributed table setup...
[INFO] Checking for blocking locks...
    SELECT pg_terminate_backend(pid)
    FROM pg_stat_activity
    WHERE pid <> pg_backend_pid()
      AND state = 'idle in transaction'
      AND query_start < NOW() - INTERVAL '30 seconds'
      AND datname = current_database();
  
 pg_terminate_backend 
----------------------
(0 rows)

[INFO] Creating demo schema (if needed)...
CREATE SCHEMA
[OK]   ✅ Demo schema ready
[INFO] Creating distributed table 'demo.events'...
CREATE TABLE
[OK]   ✅ Table is already distributed
[INFO] Inserting test data...
INSERT 0 1
[OK]   ✅ Distributed table contains 1 row(s)
[INFO] Checking shard distribution...
[OK]   ✅ Table has 1 shard(s)
[INFO] Shard placement across workers (first 10 shards):
 shardid | nodename | nodeport | shardstate 
---------+----------+----------+------------
(0 rows)
[OK]   ✅ Verified 3 shard placement(s)
[INFO] Testing query routing (EXPLAIN for user_id=42)...
[INFO]    Query plan:         QUERY PLAN        
--------------------------
 Seq Scan on events
   Filter: (user_id = 42)
(2 rows)


[INFO] 🔟 Checking data directory...
[OK]   ✅ Data directory exists: /data/postgresql/17/zone-universe-main-dev/coordinator
[OK]   ✅ Data directory size: 4.0K

[INFO] 1️⃣1️⃣ Checking PgBouncer configuration...
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/zone-universe-main-dev/pgbouncer.ini' as root on web-03.
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/%i/pgbouncer.ini' as root on web-03.
[OK]   ✅ PgBouncer is installed
[INFO]    Version: 1.24.1
2.1.12-stable
c-ares
OpenSSL
yes
[OK]   ✅ PgBouncer service is active: pgbouncer@zone-universe-main-dev.service
[WARN] ⚠️ PgBouncer IP service is not active: pgbouncer-ip@zone-universe-main-dev.service
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/zone-universe-main-dev/pgbouncer.ini' as root on web-03.
[WARN] ⚠️ PgBouncer config not found: /etc/pgbouncer/zone-universe-main-dev/pgbouncer.ini
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/zone-universe-main-dev/userlist.txt' as root on web-03.
[WARN] ⚠️ PgBouncer auth file not found: /etc/pgbouncer/zone-universe-main-dev/userlist.txt
[OK]   ✅ PgBouncer is listening on port 6432

[INFO] 1️⃣2️⃣ Enhanced PgBouncer Admin Console Verification...
[INFO] ℹ️  PgBouncer password not found

[INFO] 1️⃣3️⃣ Replicator User Connection Verification...
[INFO] ℹ️  No replication slots configured - skipping replicator verification

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ PostgreSQL verification completed successfully!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Instance:       zone-universe-main-dev-coordinator
[INFO] Service:        postgresql@zone-universe-main-dev-coordinator.service
[INFO] Socket:         /var/run/postgresql-zone-universe-main-dev-coordinator
[INFO] Data Directory: /data/postgresql/17/zone-universe-main-dev/coordinator
[INFO] Hostname:       db-zone-universe-main-dev-postgresql-coordinator.fastorder.com
[INFO] Port:           5432
[INFO] SSL:            on
[INFO] WAL Level:      logical
[INFO] Citus:          yes
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] ✅ Verification passed for coordinator


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📊 PRODUCTION READINESS CHECKS (Step 04 & 05)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔍 Checking Monitoring Setup (postgres_exporter or observability cell)...

[INFO] ℹ️  Monitoring can be configured via:
[INFO]    • Local postgres_exporter (step 04-monitoring-setup.sh)
[INFO]    • Observability Cell integration (step 02-observability-cell)

[ OK ] ✅ postgres_exporter is installed
[INFO]    Version: 0.10.1-1ubuntu0.22.04.3
[ OK ] ✅ postgres_exporter-zone-universe-main-dev.service is running
[WARN] ⚠️  Metrics endpoint not responding
[INFO] ℹ️  Monitoring user 'postgres_exporter' not found in PostgreSQL
[INFO]    This is expected if using observability cell remote monitoring
[INFO] ℹ️  Monitoring check passed (local or observability cell)

[INFO] 🔍 Checking Backup Setup (pgBackRest + WAL archiving)...

[ OK ] ✅ pgBackRest is installed
[INFO]    Version: pgBackRest 2.56.0
[ OK ] ✅ WAL archiving is enabled (archive_mode=on)
[ OK ] ✅ archive_command is configured for pgBackRest
[INFO]    Command: timeout 30 /usr/bin/pgbackrest --stanza=zone-universe-main-dev-coordinator archive-push %p
[ OK ] ✅ pgBackRest configuration exists
[ OK ] ✅ pgBackRest stanza 'zone-universe-main-dev-coordinator' is initialized
[ OK ] ✅ Backups exist (2 full backup(s))
[INFO]    Latest backup info:
                 timestamp start/stop: 2026-02-05 10:49:47+00 / 2026-02-05 10:50:01+00
                 wal start/stop: 000000010000000000000005 / 000000010000000000000005
                 database size: 55.8MB, database backup size: 55.8MB
                 repo1: backup set size: 7.9MB, backup size: 7.9MB
     
             full backup: 20260205-105023F
                 timestamp start/stop: 2026-02-05 10:50:23+00 / 2026-02-05 10:50:29+00
                 wal start/stop: 000000010000000000000008 / 000000010000000000000008
                 database size: 55.8MB, database backup size: 55.8MB
                 repo1: backup set size: 7.9MB, backup size: 7.9MB
[ OK ] ✅ Automated backup cron jobs are configured
[INFO]    Schedule:
     0 2 * * 0 root /usr/local/bin/pgbackrest-full-backup-zone-universe-main-dev.sh
     0 2 * * 1-6 root /usr/local/bin/pgbackrest-diff-backup-zone-universe-main-dev.sh
[ OK ] ✅ Backup directory exists: /var/lib/pgbackrest
[INFO]    Total backup size: 7.1G

[INFO] 🔍 Checking Worker Backup Coverage...

[INFO] ℹ️  Worker backups are optional for development environments
[INFO]    For production, ensure all workers have backup coverage

[INFO] Checking worker 1/1: worker-01...
[WARN] ⚠️  Worker worker-01 stanza exists but status unknown
[INFO] ℹ️  Incomplete worker backup coverage (0/1) - OK for dev

[INFO] 🔍 Checking Synchronous Replication (RPO=0)...

[INFO] ℹ️  Synchronous replication (RPO=0) is optional for development
[INFO]    For production with zero data loss requirement, enable sync replication

[INFO] ℹ️  Worker worker-01 synchronous replication NOT configured
[INFO]    └─ synchronous_commit: on
[INFO]    └─ synchronous_standby_names: 
[ OK ] ✅ All workers have synchronous replication (RPO=0)

[INFO] 🔍 Checking Connection and Memory Optimization...

[ OK ] ✅ Coordinator max_connections optimized: 150
[ OK ] ✅ Coordinator work_mem optimized: 8MB
[ OK ] ✅ Worker worker-01 max_connections optimized: 100
[WARN] ⚠️  Worker worker-01 work_mem not optimized: 4MB (expected: 8MB)
[WARN] ⚠️  Connection optimization incomplete

[INFO] 🔍 Checking Optimizations...

[ OK ] ✅ Citus coordinator host configured: db-zone-universe-main-dev-postgresql-coordinator.fastorder.com
[WARN] ⚠️  Periodic integrity checks NOT configured
[INFO]    Configure with: ./setup/04-postgresql/steps/04-production-optimizations.sh
[WARN] ⚠️  Backup schedule NOT staggered (all at :00)
[INFO]    Optimize with: ./setup/04-postgresql/steps/04-production-optimizations.sh
[2026-02-05 10:53:09 UTC] USER=www-data EUID=0 PID=896718 ACTION=fsop ARGS=test -f /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_zone-universe-main-dev.md
[2026-02-05 10:53:09 UTC] USER=www-data EUID=0 PID=896727 ACTION=fsop ARGS=grep -q ## Cipher Key Management /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_zone-universe-main-dev.md
[WARN] ⚠️  Cipher key management documentation missing
[INFO]    Add with: ./setup/04-postgresql/steps/04-production-optimizations.sh
[INFO] ℹ️  Offsite backup (repo2) not configured (optional for production)
[INFO]    Setup guide: ./setup/05-db/engine/postgresql/steps/14-setup-offsite-backup.sh
[WARN] ⚠️  Some production optimizations incomplete

[INFO] 🔍 Checking Citus Maintenance Daemon Health...

[INFO] Checking for stuck Citus Maintenance Daemons...
[ OK ] ✅ Citus Maintenance Daemons are healthy
[INFO] Checking for stuck distributed table operations...
[ OK ] ✅ No stuck distributed table operations
[INFO] Testing distributed table operations (10s timeout)...
[WARN] ⚠️  CRITICAL: Distributed table test TIMED OUT (10s)
[WARN]    Citus cluster is NOT operational - distributed tables cannot be created
[WARN]    This confirms maintenance daemons are stuck
[WARN]    
[WARN]    🔧 ACTION REQUIRED: Restart coordinator before using Citus
[WARN]       sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@zone-universe-main-dev-coordinator.service
[INFO] Checking for lock contention...
[ OK ] ✅ No lock contention detected
[INFO] Checking for lingering prepared transactions...
[ OK ] ✅ No lingering prepared transactions

[WARN] ⚠️  Citus cluster has health issues - see warnings above
[WARN]    
[WARN]    ⚡ IMMEDIATE ACTION: Restart coordinator to restore Citus functionality
[WARN]       sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@zone-universe-main-dev.service

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📋 PRODUCTION READINESS SUMMARY
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Component                 Status          Production Ready?   
───────────────────────── ─────────────── ────────────────────
Citus Cluster             ✅ Operational YES                 
High Availability         ✅ Configured  YES                 
SSL/TLS Security          ✅ Enabled     YES                 
PgBouncer                 ✅ Running     YES                 
Monitoring                ✅ Operational YES                 
Backups (Coordinator)     ✅ Configured  YES                 
Backups (Workers)         ✅ Configured  YES                 
Sync Replication (RPO=0)  ✅ Enabled     YES                 
Connection Optimization   ⚠️  Not Ready NO                  
Optimizations             ⚠️  Incomplete OPTIONAL            

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[ OK ] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] 🎉 PRODUCTION READY: 100% (3/3 critical checks passed)
[ OK ] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ All critical components are operational and production-ready

[INFO] Next steps:
[INFO]   1. Configure Prometheus to scrape metrics: http://localhost:9196/metrics
[INFO]   2. Import Grafana dashboards for PostgreSQL + Citus monitoring
[INFO]   3. Setup alerting rules for critical metrics
[INFO]   4. Schedule regular restore drills (monthly)
[INFO]   5. Review /var/www/html/skeleton.dev.fastorder.com/fixing/scripts/PRODUCTION_READINESS.md

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✓ Verification process completed successfully
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 18-production-optimizations.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PostgreSQL Production Optimizations
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Environment: zone-universe-main-dev
[INFO] Enable Sync Replication: --auto

[INFO] 1️⃣ Configuring Citus coordinator hostname...
[ OK ] ✅ Coordinator hostname already configured: db-zone-universe-main-dev-postgresql-coordinator.fastorder.com

[INFO] 2️⃣ Configuring synchronous replication for RPO=0...
[INFO] Synchronous replication NOT enabled (use './04-production-optimizations.sh yes' to enable)
[INFO] Current configuration: async replication (RPO > 0)
[INFO] 
[INFO] To enable safely after deployment:
[INFO]   /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/enable_sync_replication_safe.sh \
[INFO]     /var/run/postgresql-zone-universe-main-dev-worker-01 worker_01_standby_01

[INFO] 3️⃣ Adding periodic integrity check cron jobs...
[2026-02-05 10:53:12 UTC] USER=www-data EUID=0 PID=896968 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-integrity-zone-universe-main-dev
[ OK ] ✅ Integrity check cron jobs configured
[INFO]    Daily checks: 02:15, 03:30, 04:45 (coordinator, worker-01, worker-02)
[INFO]    Weekly verify: Sundays at same times

[INFO] 4️⃣ Updating backup schedule with staggered timing...
[ OK ] ✅ Backup schedule staggered:
[INFO]    Coordinator: 02:05 (full: Sun, diff: Mon-Sat)
[INFO]    Worker-01:   03:10 (full: Sun, diff: Mon-Sat)
[INFO]    Worker-02:   04:15 (full: Sun, diff: Mon-Sat)

[INFO] 5️⃣ Documenting cipher key backup procedures...
[2026-02-05 10:53:12 UTC] USER=www-data EUID=0 PID=897081 ACTION=fsop ARGS=test -f /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_zone-universe-main-dev.md
Sorry, user www-data is not allowed to execute '/usr/bin/grep -q ## Cipher Key Management /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_zone-universe-main-dev.md' as root on web-03.
[ OK ] ✅ Cipher key documentation added to /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_zone-universe-main-dev.md

[INFO] 6️⃣ Checking offsite backup configuration...
[INFO] ℹ️  Offsite backup (repo2) is NOT configured
[INFO]    Configuration example: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[INFO]    Setup instructions: ./setup/04-postgresql/steps/08-setup-offsite-backup.sh
[ OK ] ✅ Offsite backup example available: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Production Optimizations Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[ OK ] Applied optimizations:
[ OK ]   1. ✅ Citus coordinator hostname
[ OK ]   2. ⏭️  Synchronous replication (RPO=0)
[ OK ]   3. ✅ Periodic integrity checks (daily + weekly)
[ OK ]   4. ✅ Staggered backup schedule (reduced load spikes)
[ OK ]   5. ✅ Cipher key backup documentation
[ OK ]   6. ✅ Offsite backup (repo2) example configuration

[INFO] Next steps:
[INFO]   1. Backup cipher keys to secure vault immediately
[INFO]   2. Set up S3/MinIO for offsite backups:
[INFO]      - Instructions: ./setup/04-postgresql/steps/08-setup-offsite-backup.sh
[INFO]      - Example config: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[INFO]   3. Configure alerting for backup failures (cron exit codes)
[INFO]   4. Test restore drill from offsite repository
[INFO]   5. Enable RPO=0 if needed: ./04-production-optimizations.sh yes

[ OK ] System is now production-grade! 🎉

✓ ✔ PostgreSQL creation completed
[INFO] Cleaning up temporary files...
[INFO] Starting cleanup of temporary files...
[INFO] Cleaning up SSL temp files for zone-universe-main-dev...
[INFO] Cleaning up old provisioning logs...
[INFO] Cleaning up old configuration backups...
✓ ✔ Cleanup completed

✓ ✅ Database infrastructure (postgresql) setup completed successfully

[INFO] Using environment from web interface: zone-universe-main-dev
[2026-02-05 10:53:14] Using web-provided environment: zone-universe-main-dev
[2026-02-05 10:53:14] Service: zone, Zone: universe, Branch: main, Env: dev
✓ Environment initialized successfully (mode: general)
[INFO] Starting finalizing setup process...
[INFO] Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps
[INFO] Environment: zone-universe-main-dev

[INFO] Found 3 step(s) to execute

[INFO] 📦 Step 1/3: enable_disable_all_applications...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
════════════════════════════════════════════════════════════════════════════════
  Environment Services Management
════════════════════════════════════════════════════════════════════════════════
  Environment:  zone-universe-main-dev
  Action:       enable
  Triggered by: false
════════════════════════════════════════════════════════════════════════════════

🔍 Scanning for environment-specific services...
✅ Found 7 services for environment: zone-universe-main-dev

📋 Services to enable:
────────────────────────────────────────────────────────────────────────────────
  • confluent-connect-zone-universe-main-dev_coordinator.service [active/unmasked/enabled]
  • confluent-kraft-zone-universe-main-dev_coordinator.service   [active/unmasked/enabled]
  • elasticsearch@zone-universe-main-dev-node-01.service         [active/unmasked/enabled]
  • pgbouncer@zone-universe-main-dev.service                     [active/unmasked/enabled]
  • postgresql@zone-universe-main-dev-coordinator.service        [active/unmasked/enabled]
  • postgresql@zone-universe-main-dev-worker-01-standby-01.service [active/unmasked/enabled]
  • postgresql@zone-universe-main-dev-worker-01.service          [active/unmasked/enabled]
────────────────────────────────────────────────────────────────────────────────


❌ Cancelled by user
[OK] ✅ Step 1 completed: 01-enable_disable_all_applications.sh

[INFO] 📦 Step 2/3: verify monitoring...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Monitoring Verification for zone-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Detecting installed services...
Failed to print table: Broken pipe
[OK] ✓ PostgreSQL detected
Failed to print table: Broken pipe
[OK] ✓ Elasticsearch detected
Failed to print table: Broken pipe
[OK] ✓ Kafka detected
Failed to print table: Broken pipe
[OK] ✓ PgBouncer detected

[INFO] Services to verify: postgresql elasticsearch kafka pgbouncer

[INFO] 2️⃣ Verifying exporters are running...
[OK] ✓ PostgreSQL exporter is running
[OK] ✓ Elasticsearch exporter is running
[OK] ✓ Kafka JMX exporter is running
[OK] ✓ PgBouncer exporter is running

[INFO] 3️⃣ Verifying Prometheus configuration...
[2026-02-05 10:53:16 UTC] USER=www-data EUID=0 PID=897604 ACTION=passthru ARGS=grep -q job_name: 'postgresql' /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml
[OK] ✓ postgresql is configured in Prometheus
[2026-02-05 10:53:17 UTC] USER=www-data EUID=0 PID=897627 ACTION=passthru ARGS=grep -q job_name: 'elasticsearch' /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml
[OK] ✓ elasticsearch is configured in Prometheus
[2026-02-05 10:53:17 UTC] USER=www-data EUID=0 PID=897649 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml
[WARN] ⚠️  kafka is not configured in Prometheus scrape targets
[WARN] ⚠️  pgbouncer is not configured in Prometheus scrape targets

[INFO] 4️⃣ Verifying Prometheus is actively scraping...
[OK] ✓ Prometheus is running
[OK] ✓ postgresql target is UP in Prometheus
[OK] ✓ elasticsearch target is UP in Prometheus
[WARN] ⚠️  kafka target is not UP in Prometheus (may still be initializing)
[WARN] ⚠️  pgbouncer target is not UP in Prometheus (may still be initializing)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Monitoring Verification Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[WARN] Some monitoring issues were detected:

[WARN] Prometheus Configuration Issues:
  - kafka not configured in Prometheus
  - pgbouncer not configured in Prometheus

[WARN] Automatically running monitoring setup scripts to fix issues...

[INFO] Running Kafka monitoring setup...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Kafka Monitoring Integration for zone-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-zone-universe-main-dev
[OK]   Observability cell endpoints registered for zone-universe-main-dev
[OK]   ✓ Observability cell is ready

[INFO] 2️⃣ Setting up Kafka JMX exporter integration...
[INFO] JMX Exporter port calculated for zone-universe-main-dev: 9403 (offset: 95)
[INFO] Checking observability cell readiness: obs-zone-universe-main-dev
[OK]   Observability cell endpoints registered for zone-universe-main-dev
[INFO] Setting up Kafka JMX exporter for zone-universe-main-dev
[INFO] JMX Prometheus Java Agent already exists at /opt/kafka/libs/jmx_prometheus_javaagent.jar
[2026-02-05 10:53:22 UTC] USER=www-data EUID=0 PID=897969 ACTION=passthru ARGS=mv /tmp/jmx_exporter.yml /opt/kafka/config/jmx_exporter.yml
[OK]   JMX exporter configuration created at /opt/kafka/config/jmx_exporter.yml
[OK]   JMX exporter configuration created
[INFO] Configuring Kafka systemd services to use JMX exporter...
[2026-02-05 10:53:23 UTC] USER=www-data EUID=0 PID=898007 ACTION=fsop ARGS=test -f /etc/systemd/system/[2026-02-05
[INFO] All Kafka services already configured with JMX exporter
[OK]   Kafka JMX exporter integration complete
[INFO] Metrics endpoint: http://142.93.238.16:9403/metrics
[INFO] Prometheus will automatically scrape: https://metrics-zone-universe-main-dev.fastorder.com:9090
[INFO] View dashboards at: https://dashboards-zone-universe-main-dev.fastorder.com
[OK]   ✓ Kafka JMX exporter integration complete
[INFO] Configuring KAFKA_OPTS environment variable for kafka user...
[2026-02-05 10:53:23 UTC] USER=www-data EUID=0 PID=898031 ACTION=passthru ARGS=grep -q KAFKA_OPTS.*javaagent.*jmx_prometheus_javaagent.*=9403: /home/kafka/.bashrc
[2026-02-05 10:53:23 UTC] USER=www-data EUID=0 PID=898065 ACTION=passthru ARGS=sed -i /export KAFKA_OPTS=.*jmx_prometheus_javaagent/d /home/kafka/.bashrc
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[OK]   ✓ KAFKA_OPTS configured in /home/kafka/.bashrc (port 9403)
[INFO] 2.5️⃣ Enabling JMX exporter in Kafka systemd service...
[2026-02-05 10:53:23 UTC] USER=www-data EUID=0 PID=898116 ACTION=passthru ARGS=grep -q javaagent.*jmx_prometheus_javaagent /etc/systemd/system/confluent-kraft-zone-universe-main-dev_coordinator.service
[OK]   ✓ JMX exporter already enabled in Kafka systemd services
[INFO] 2.6️⃣ Configuring Prometheus to scrape Kafka metrics...
[2026-02-05 10:53:23 UTC] USER=www-data EUID=0 PID=898142 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml
[INFO] Adding Kafka scrape target to Prometheus configuration...
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[2026-02-05 10:53:23 UTC] USER=www-data EUID=0 PID=898177 ACTION=passthru ARGS=sed -i /# Prometheus self-monitoring/r /tmp/prometheus_kafka_add.yml /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml
[ERROR] Invalid Prometheus configuration - rolling back
[2026-02-05 10:53:23 UTC] USER=www-data EUID=0 PID=898212 ACTION=passthru ARGS=sed -i /job_name: 'kafka'/,+6d /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml
[2026-02-05 10:53:23 UTC] USER=www-data EUID=0 PID=898265 ACTION=fsop ARGS=rm -f /tmp/prometheus_kafka_add.yml

[INFO] 3️⃣ Registering Kafka nodes to monitoring database...
[INFO] Detected Kafka version: 3.9.1
[INFO] Registering Kafka Broker to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Kafka Broker
[INFO]   Identifier:        zone-universe-main-dev-broker-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                142.93.238.16
[INFO]   Port:              9092
[INFO]   FQDN:              eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 447d4dd5-e271-4d9e-91d6-bc2f569019ee
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   ✓ Kafka broker registered
[INFO] Registering Kafka Connect to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Kafka Connect
[INFO]   Identifier:        zone-universe-main-dev-connect-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8083
[INFO]   FQDN:              eventbus-zone-universe-main-dev-kafka-connect.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 23c140b3-b995-4158-a535-f8c0a04adfa6
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   ✓ Kafka Connect registered
[INFO] Schema Registry not running, skipping registration

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Kafka Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Metrics: http://localhost:9403/metrics
[INFO] Prometheus: https://metrics-zone-universe-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-zone-universe-main-dev.fastorder.com
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✓ Kafka monitoring setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Step 2 completed: 02-verify-monitoring.sh

[INFO] 📦 Step 3/3: register backup infrastructure...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔧 Registering Core Services & Backup Infrastructure for zone-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Registering Main App...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Main App
[INFO]   Identifier:        zone-universe-main-dev-main-app
[INFO]   Identifier Parent: application
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8080
[INFO]   FQDN:              app-zone-universe-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: f5c567f1-ba97-438d-a46c-32b494b1e8e0
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
/opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps/03-register-backup-infrastructure.sh: line 70: ok: command not found

[INFO] 2️⃣ Registering Audit Service...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Audit Service
[INFO]   Identifier:        zone-universe-main-dev-audit
[INFO]   Identifier Parent: application
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8081
[INFO]   FQDN:              audit-zone-universe-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: e1ba1eb5-d90d-4d32-bf34-31cf89ac1f09
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
/opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps/03-register-backup-infrastructure.sh: line 85: ok: command not found

[INFO] 3️⃣ Registering PostgreSQL Backup Node...
[ERROR] Invalid identifier format: backup-db
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register PostgreSQL backup node (non-blocking)

[INFO] 4️⃣ Registering Elasticsearch Backup Node...
[ERROR] Invalid identifier format: backup-search
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Elasticsearch backup node (non-blocking)

[INFO] 5️⃣ Registering Kafka Backup Node...
[ERROR] Invalid identifier format: backup-eventbus
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Kafka backup node (non-blocking)

[INFO] 6️⃣ Registering Backup Orchestrator...
[ERROR] Invalid identifier format: backup-orchestrator
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Backup orchestrator (non-blocking)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Core Services & Backup Infrastructure Registration Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Registered core services:
[INFO]   🚀 main-app            → Core application service
[INFO]   📋 audit               → Centralized audit logging (WORM)

[INFO] Registered backup nodes:
[INFO]   📦 backup-db           → PostgreSQL backup (pgBackRest, PITR)
[INFO]   📦 backup-search       → Elasticsearch snapshots (ILM, S3)
[INFO]   📦 backup-eventbus     → Kafka log segments (replication)
[INFO]   📦 backup-orchestrator → Central backup coordination

[INFO] Dashboard: https://skeleton.dev.fastorder.com/dashboard/monitoring
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Step 3 completed: 03-register-backup-infrastructure.sh


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ finalizing setup completed successfully!
[OK] Executed all 3 steps
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Environment: zone-universe-main-dev
[INFO] Service: zone
[INFO] Zone: universe
[INFO] Branch: main
[INFO] Env: dev