Environment: Web Universe Main Dev on web-03
"{\"env\": \"dev\", \"zone\": \"universe\", \"branch\": \"main\", \"db_app\": \"postgresql\", \"service\": \"web\", \"db_enabled\": true, \"pg_standby\": 0, \"pg_workers\": 1, \"description\": \"\", \"iam_enabled\": false, \"worker_1_ip\": \"\", \"eventbus_app\": \"kafka\", \"worker_1_fqdn\": \"db-web-universe-main-dev-postgresql-worker-01.fastorder.com\", \"search_enabled\": false, \"eventbus_enabled\": true, \"postgresql_enabled\": true, \"postgresql_run_verification\": true}"
This job encountered an error. You can restart from the failed step.
This job has been restarted. You are viewing an older attempt. The logs and status shown below are from the latest retry.
This job failed at one of the steps below. You can resume from where it failed to save time and avoid re-running successful steps.
⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...[0;34m[INFO][0m Using eventbus engine from EVENTBUS_ENGINE environment variable: kafka
[0;34m[INFO][0m Cleaning up any existing locks...
[0;32m[1mStarting eventbus engine: kafka[0m
[1;33m═══════════════════════════════════════════════[0m
[0;34m[INFO][0m Using environment from web interface: web-universe-main-dev
[0;32m[2026-02-05 04:44:05][0m Using web-provided environment: web-universe-main-dev
[0;32m[2026-02-05 04:44:05][0m Service: web, Zone: universe, Branch: main, Env: dev
[0;32m✓[0m Environment initialized successfully (mode: general)
[0;34m[INFO][0m Starting Kafka setup process...
[0;34m[INFO][0m Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Found 13 step(s) to execute
[0;34m[INFO][0m 📦 Step 1/13: install debezium connector...
═══════════════════════════════════════════════════════════════════
Fetching latest versions from Maven Central...
Installing Debezium PostgreSQL Connector
Debezium version: 3.4.1.Final
pgjdbc version: 42.7.9
═══════════════════════════════════════════════════════════════════
[OK] Debezium 3.4.1.Final with pgjdbc 42.7.9 already installed
[0;32m[OK][0m ✅ Step 1 completed: 00-install-debezium-connector.sh
[0;34m[INFO][0m 📦 Step 2/13: kafka setup...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials for secrets vault...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
🧹 Checking for orphaned Kafka processes on ports 9092, 9093, 8083...
⚠️ Found process on port 9092 (PIDs: [2026-02-05 04:44:06 UTC] USER=www-data EUID=0 PID=3720375 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 2>/dev/null || true
3593645
3594895
3594896
3594897
3594898
3594899
3598756
3606773), killing...
[2026-02-05 04:44:06 UTC] USER=www-data EUID=0 PID=3720390 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 04:44:06 UTC] USER=www-data EUID=0 PID=3720375 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 2>/dev/null || true
3593645
3594895
3594896
3594897
3594898
3594899
3598756
3606773 2>/dev/null || true
/usr/bin/bash: line 2: 3593645: command not found
/usr/bin/bash: line 3: 3594895: command not found
/usr/bin/bash: line 4: 3594896: command not found
/usr/bin/bash: line 5: 3594897: command not found
/usr/bin/bash: line 6: 3594898: command not found
/usr/bin/bash: line 7: 3594899: command not found
/usr/bin/bash: line 8: 3598756: command not found
⚠️ Found process on port 9093 (PIDs: [2026-02-05 04:44:07 UTC] USER=www-data EUID=0 PID=3720422 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 2>/dev/null || true
3594698
3594895
3594897
3606773), killing...
[2026-02-05 04:44:08 UTC] USER=www-data EUID=0 PID=3720445 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 04:44:07 UTC] USER=www-data EUID=0 PID=3720422 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 2>/dev/null || true
3594698
3594895
3594897
3606773 2>/dev/null || true
/usr/bin/bash: line 2: 3594698: command not found
/usr/bin/bash: line 3: 3594895: command not found
/usr/bin/bash: line 4: 3594897: command not found
⚠️ Found process on port 8083 (PIDs: [2026-02-05 04:44:09 UTC] USER=www-data EUID=0 PID=3720485 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 2>/dev/null || true
3593645
3594896
3594898), killing...
[2026-02-05 04:44:09 UTC] USER=www-data EUID=0 PID=3720497 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 04:44:09 UTC] USER=www-data EUID=0 PID=3720485 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 2>/dev/null || true
3593645
3594896
3594898 2>/dev/null || true
/usr/bin/bash: line 2: 3593645: command not found
/usr/bin/bash: line 3: 3594896: command not found
✅ Port cleanup completed
Ensuring KAFKA application environment for coordinator...
[0;34m[INFO][0m Using existing KAFKA environment: eventbus-web-universe-main-dev-kafka-connect.fastorder.com (10.100.1.75)
Ensuring KAFKA_BROKER_IP application environment for coordinator...
[0;34m[INFO][0m Using existing KAFKA BROKER environment: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com (10.100.1.225)
[0;34m[INFO][0m Kafka Broker IP: 10.100.1.225
[0;34m[INFO][0m Kafka Connect IP: 10.100.1.75
[0;34m[INFO][0m Registered /etc/hosts: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com -> 10.100.1.225
[0;34m[INFO][0m Registered /etc/hosts: eventbus-web-universe-main-dev-kafka-connect.fastorder.com -> 10.100.1.75
🔐 Initializing keystore passwords...
[0;34m[INFO][0m 🔍 Checking secrets backend (provider: aws)...
[0;32m✅ Retrieved passwords from remote backend[0m
[0;34m[INFO][0m ✅ Using existing passwords from backend
✅ Keystore passwords initialized
- Keystore password: HGvJkWmj... (32 chars)
- Truststore password: sZRdI2nT... (32 chars)
[0;34m[INFO][0m 🔐 Vaulting kafka passwords to remote backend...
[0;32m✅ Passwords vaulted to remote backend[0m
✅ Kafka keystore passwords saved to AWS Secrets Manager
[INFO] Generating for: web-universe-main-dev (host=eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com ip=10.100.1.225)
[2026-02-05 04:44:15 UTC] USER=www-data EUID=0 PID=3720758 ACTION=fsop ARGS=rm -rf /opt/kafka/secrets/web-universe-main-dev/coordinator /data/kafka/web-universe-main-dev/coordinator
[2026-02-05 04:44:15 UTC] USER=www-data EUID=0 PID=3720767 ACTION=fsop ARGS=mkdir -p /opt/kafka/secrets/web-universe-main-dev/coordinator /opt/kafka/config/web-universe-main-dev/coordinator /opt/kafka/secrets/web-universe-main-dev/coordinator/pem /data/kafka/web-universe-main-dev_coordinator-data
[2026-02-05 04:44:15 UTC] USER=www-data EUID=0 PID=3720780 ACTION=fsop ARGS=chown -R kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator
[2026-02-05 04:44:15 UTC] USER=www-data EUID=0 PID=3720793 ACTION=fsop ARGS=chown -R kafka:kafka /opt/kafka/config/web-universe-main-dev/coordinator /data/kafka/web-universe-main-dev_coordinator-data
[2026-02-05 04:44:15 UTC] USER=www-data EUID=0 PID=3720825 ACTION=fsop ARGS=chmod 770 /opt/kafka/config/web-universe-main-dev/coordinator /data/kafka/web-universe-main-dev_coordinator-data
[2026-02-05 04:44:15 UTC] USER=www-data EUID=0 PID=3720834 ACTION=fsop ARGS=chmod 750 /opt/kafka/secrets/web-universe-main-dev/coordinator
[2026-02-05 04:44:15 UTC] USER=www-data EUID=0 PID=3720843 ACTION=fsop ARGS=chmod 750 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
[2026-02-05 04:44:15 UTC] USER=www-data EUID=0 PID=3720855 ACTION=fsop ARGS=chmod 700 /tmp/fo-tls.jjpNsy
[2026-02-05 04:44:15 UTC] USER=www-data EUID=0 PID=3720864 ACTION=fsop ARGS=chmod 755 /tmp/fo-tls.jjpNsy
[2026-02-05 04:44:15 UTC] USER=www-data EUID=0 PID=3720873 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/fo-tls.jjpNsy/ra_root.crt
[2026-02-05 04:44:15 UTC] USER=www-data EUID=0 PID=3720882 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/fo-tls.jjpNsy/ra_root.key
[2026-02-05 04:44:15 UTC] USER=www-data EUID=0 PID=3720892 ACTION=fsop ARGS=chmod 644 /tmp/fo-tls.jjpNsy/ra_root.crt
[2026-02-05 04:44:15 UTC] USER=www-data EUID=0 PID=3720901 ACTION=fsop ARGS=chmod 644 /tmp/fo-tls.jjpNsy/ra_root.key
Certificate was added to keystore
[2026-02-05 04:44:16 UTC] USER=www-data EUID=0 PID=3720935 ACTION=fsop ARGS=mv /tmp/fo-tls.jjpNsy/truststore.jks /opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks
[2026-02-05 04:44:16 UTC] USER=www-data EUID=0 PID=3720944 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks
[2026-02-05 04:44:16 UTC] USER=www-data EUID=0 PID=3720955 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks
Generating 4,096 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 825 days
for: CN=eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com, OU=Kafka Broker, O=FastOrder, C=AE
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /tmp/fo-tls.jjpNsy/kafka.server.keystore.jks -destkeystore /tmp/fo-tls.jjpNsy/kafka.server.keystore.jks -deststoretype pkcs12".
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /tmp/fo-tls.jjpNsy/kafka.server.keystore.jks -destkeystore /tmp/fo-tls.jjpNsy/kafka.server.keystore.jks -deststoretype pkcs12".
Certificate request self-signature ok
subject=C = AE, O = FastOrder, OU = Kafka Broker, CN = eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com
Certificate was added to keystore
Warning:
Certificate reply was installed in keystore
Warning:
[2026-02-05 04:44:21 UTC] USER=www-data EUID=0 PID=3721248 ACTION=fsop ARGS=mv /tmp/fo-tls.jjpNsy/kafka.server.keystore.jks /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 04:44:21 UTC] USER=www-data EUID=0 PID=3721257 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 04:44:21 UTC] USER=www-data EUID=0 PID=3721266 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.server.keystore.jks
Generating 4,096 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 825 days
for: CN=eventbus-web-universe-main-dev-kafka-connect.fastorder.com, OU=Kafka Connect REST, O=FastOrder, C=AE
Certificate request self-signature ok
subject=C = AE, O = FastOrder, OU = Kafka Connect REST, CN = eventbus-web-universe-main-dev-kafka-connect.fastorder.com
Certificate was added to keystore
Certificate reply was installed in keystore
[2026-02-05 04:44:27 UTC] USER=www-data EUID=0 PID=3721474 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/connect-rest.keystore.p12
Certificate request self-signature ok
subject=CN = kafka-client-web-universe-main-dev, OU = Kafka Client, O = FastOrder, C = AE
[2026-02-05 04:44:28 UTC] USER=www-data EUID=0 PID=3721504 ACTION=fsop ARGS=cp /tmp/fo-tls.jjpNsy/ra_root.crt /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem
[2026-02-05 04:44:28 UTC] USER=www-data EUID=0 PID=3721515 ACTION=fsop ARGS=cp /tmp/fo-tls.jjpNsy/client-key.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:44:28 UTC] USER=www-data EUID=0 PID=3721524 ACTION=fsop ARGS=cp /tmp/fo-tls.jjpNsy/client-cert.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
[2026-02-05 04:44:28 UTC] USER=www-data EUID=0 PID=3721540 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
[2026-02-05 04:44:28 UTC] USER=www-data EUID=0 PID=3721552 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
[2026-02-05 04:44:28 UTC] USER=www-data EUID=0 PID=3721563 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:44:28 UTC] USER=www-data EUID=0 PID=3721582 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.client.keystore.p12
🔐 Ensuring kafka user has access to PostgreSQL certificates...
✅ kafka is already in postgres group
🧹 Cleaning up conflicting services and processes on Kafka ports on 10.100.1.225...
🔪 Killing processes on 10.100.1.225:8083: [2026-02-05 04:44:28 UTC] USER=www-data EUID=0 PID=3721639 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:8083 -t 2>/dev/null || true
[2026-02-05 04:44:29 UTC] USER=www-data EUID=0 PID=3721658 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 04:44:28 UTC] USER=www-data EUID=0 PID=3721639 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:8083 -t 2>/dev/null || true
🔪 Killing processes on 10.100.1.225:9092: [2026-02-05 04:44:29 UTC] USER=www-data EUID=0 PID=3721669 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:9092 -t 2>/dev/null || true
3606773
[2026-02-05 04:44:29 UTC] USER=www-data EUID=0 PID=3721688 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 04:44:29 UTC] USER=www-data EUID=0 PID=3721669 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:9092 -t 2>/dev/null || true
3606773
/usr/bin/bash: line 2: 3606773: command not found
🔪 Killing processes on 10.100.1.225:9093: [2026-02-05 04:44:29 UTC] USER=www-data EUID=0 PID=3721701 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:9093 -t 2>/dev/null || true
3606773
[2026-02-05 04:44:30 UTC] USER=www-data EUID=0 PID=3721725 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 04:44:29 UTC] USER=www-data EUID=0 PID=3721701 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:9093 -t 2>/dev/null || true
3606773
/usr/bin/bash: line 2: 3606773: command not found
✅ Port cleanup completed
🔧 Checking for Kafka Connect internal topics with incorrect cleanup policy...
📋 Kafka broker is running, checking topic cleanup policies...
✅ Topic cleanup policy fix completed
🔧 Creating environment-specific systemd units...
🔧 Writing client properties to /etc/kafka/client-web-universe-main-dev-coordinator.properties ...
[2026-02-05 04:44:42 UTC] USER=www-data EUID=0 PID=3723296 ACTION=fsop ARGS=chown root:kafka /etc/kafka/client-web-universe-main-dev-coordinator.properties
[2026-02-05 04:44:42 UTC] USER=www-data EUID=0 PID=3723310 ACTION=fsop ARGS=chmod 0644 /etc/kafka/client-web-universe-main-dev-coordinator.properties
🔧 Creating PEM certificates for PHP mTLS access...
[2026-02-05 04:44:42 UTC] USER=www-data EUID=0 PID=3723319 ACTION=passthru ARGS=bash -c openssl pkcs12 -in '/opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.client.keystore.p12' -clcerts -nokeys -passin pass:'HGvJkWmjjIaZzWVQzIjopYiQoGhZCsRH' -out '/opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.crt' 2>/dev/null
[2026-02-05 04:44:42 UTC] USER=www-data EUID=0 PID=3723329 ACTION=passthru ARGS=bash -c openssl pkcs12 -in '/opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.client.keystore.p12' -nocerts -nodes -passin pass:'HGvJkWmjjIaZzWVQzIjopYiQoGhZCsRH' -out '/opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.key' 2>/dev/null
[2026-02-05 04:44:42 UTC] USER=www-data EUID=0 PID=3723351 ACTION=passthru ARGS=bash -c keytool -exportcert -alias fastorder-ra-root -keystore '/opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks' -storepass 'sZRdI2nTnX4yhzopYtu0ttl9GtAWGH7c' -rfc -file '/opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.crt' 2>/dev/null
[2026-02-05 04:44:43 UTC] USER=www-data EUID=0 PID=3723386 ACTION=fsop ARGS=chown root:kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.crt /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.key /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.crt
[2026-02-05 04:44:43 UTC] USER=www-data EUID=0 PID=3723395 ACTION=fsop ARGS=chmod 0644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.crt /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.crt
[2026-02-05 04:44:43 UTC] USER=www-data EUID=0 PID=3723404 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.key
🔐 Creating connector secrets file for FileConfigProvider...
[2026-02-05 04:44:43 UTC] USER=www-data EUID=0 PID=3723438 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/connector-secrets.properties
[2026-02-05 04:44:43 UTC] USER=www-data EUID=0 PID=3723447 ACTION=fsop ARGS=chmod 0600 /opt/kafka/secrets/web-universe-main-dev/coordinator/connector-secrets.properties
✅ Connector secrets file created: /opt/kafka/secrets/web-universe-main-dev/coordinator/connector-secrets.properties
FileConfigProvider syntax: ${file:/opt/kafka/secrets/web-universe-main-dev/coordinator/connector-secrets.properties:key_name}
🔧 Creating Canary Event timer for pipeline verification...
[2026-02-05 04:44:43 UTC] USER=www-data EUID=0 PID=3723474 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 04:44:44 UTC] USER=www-data EUID=0 PID=3723525 ACTION=passthru ARGS=systemctl enable kafka-canary-web-universe-main-dev.timer
[2026-02-05 04:44:44 UTC] USER=www-data EUID=0 PID=3723581 ACTION=passthru ARGS=systemctl start kafka-canary-web-universe-main-dev.timer
✅ Canary timer installed: kafka-canary-web-universe-main-dev.timer (every 5 minutes)
[2026-02-05 04:44:45 UTC] USER=www-data EUID=0 PID=3723591 ACTION=passthru ARGS=systemctl daemon-reload
Failed to print table: Broken pipe
[2026-02-05 04:44:52 UTC] USER=www-data EUID=0 PID=3723728 ACTION=passthru ARGS=systemctl mask kafka-broker
🔒 Adjusting group ownership and permissions ...
[2026-02-05 04:44:53 UTC] USER=www-data EUID=0 PID=3723862 ACTION=fsop ARGS=chown :kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 04:44:53 UTC] USER=www-data EUID=0 PID=3723901 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.client.keystore.p12
[2026-02-05 04:44:53 UTC] USER=www-data EUID=0 PID=3723924 ACTION=fsop ARGS=chown root:kafka /etc/kafka/client-web-universe-main-dev-coordinator.properties
[2026-02-05 04:44:53 UTC] USER=www-data EUID=0 PID=3723942 ACTION=fsop ARGS=chmod 0644 /etc/kafka/client-web-universe-main-dev-coordinator.properties
✅ Kafka configuration complete for web-universe-main-dev_coordinator
Broker ID : 89
Broker keystore : /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.server.keystore.jks
REST keystore : /opt/kafka/secrets/web-universe-main-dev/coordinator/connect-rest.keystore.p12
Truststore : /opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks
Client PKCS12 : /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.client.keystore.p12
Data directory : /data/kafka/web-universe-main-dev_coordinator-data
Server config : /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
Connect config : /opt/kafka/config/web-universe-main-dev/coordinator/connect-distributed.properties
CLI client config : /etc/kafka/client-web-universe-main-dev-coordinator.properties
🎯 Next step: Run 03-restart-kafka-related-services.sh to start services
[0;32m[OK][0m ✅ Step 2 completed: 01-kafka-setup.sh
[0;34m[INFO][0m 📦 Step 3/13: metadata...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 🎯 Kafka metadata mode: kraft
╔════════════════════════════════════════════════════════════════════╗
║ Kafka Metadata Layer Setup ║
╚════════════════════════════════════════════════════════════════════╝
Environment : web-universe-main-dev
Service : web
Zone : universe
Branch : main
Environment : dev
VM IP : 142.93.238.16
Metadata Mode : kraft
📋 KRaft Mode (Modern)
────────────────────────────────────────────────────────────────
✅ No ZooKeeper dependency
✅ Faster metadata operations
✅ Simplified architecture
✅ Recommended for new deployments
⚠️ Requires Kafka 3.3+ in production
════════════════════════════════════════════════════════════════════
[INFO] 🚀 Executing KRaft setup script...
[INFO] Script: /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps/metadata/kraft.sh
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 04:44:54 UTC] USER=www-data EUID=0 PID=3723996 ACTION=fsop ARGS=mkdir -p /data/kafka/web-universe-main-dev_coordinator-meta /opt/kafka/config/web-universe-main-dev/coordinator /data/kafka/web-universe-main-dev_coordinator-data
[2026-02-05 04:44:54 UTC] USER=www-data EUID=0 PID=3724005 ACTION=fsop ARGS=chown -R kafka:kafka /data/kafka/web-universe-main-dev_coordinator-meta /opt/kafka/config/web-universe-main-dev/coordinator /data/kafka/web-universe-main-dev_coordinator-data
[2026-02-05 04:44:54 UTC] USER=www-data EUID=0 PID=3724015 ACTION=fsop ARGS=chmod 770 /data/kafka/web-universe-main-dev_coordinator-meta /opt/kafka/config/web-universe-main-dev/coordinator /data/kafka/web-universe-main-dev_coordinator-data
[INFO] Adding eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com to /etc/hosts -> 10.100.1.225
[INFO] Adding eventbus-web-universe-main-dev-kafka-connect.fastorder.com to /etc/hosts -> 10.100.1.75
[INFO] Setting up KRaft for: web-universe-main-dev (host=eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com ip=10.100.1.225)
[2026-02-05 04:44:54 UTC] USER=www-data EUID=0 PID=3724039 ACTION=fsop ARGS=mkdir -p /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev
[INFO] Using existing cluster.id from state
🔧 Configuring Kafka for KRaft mode...
[2026-02-05 04:44:54 UTC] USER=www-data EUID=0 PID=3724057 ACTION=fsop ARGS=test -f /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 04:44:54 UTC] USER=www-data EUID=0 PID=3724066 ACTION=fsop ARGS=test -r /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 04:44:54 UTC] USER=www-data EUID=0 PID=3724075 ACTION=fsop ARGS=sed -i /^zookeeper\.connect=/d /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 04:44:54 UTC] USER=www-data EUID=0 PID=3724084 ACTION=passthru ARGS=bash -c grep -q '^process.roles=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 04:44:54 UTC] USER=www-data EUID=0 PID=3724103 ACTION=passthru ARGS=bash -c grep -q '^node.id=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 04:44:54 UTC] USER=www-data EUID=0 PID=3724122 ACTION=passthru ARGS=bash -c grep -q '^broker.id=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 04:44:54 UTC] USER=www-data EUID=0 PID=3724131 ACTION=fsop ARGS=sed -i s|^broker.id=.*|broker.id=1| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 04:44:54 UTC] USER=www-data EUID=0 PID=3724140 ACTION=passthru ARGS=bash -c grep -q '^controller.listener.names=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 04:44:54 UTC] USER=www-data EUID=0 PID=3724159 ACTION=passthru ARGS=bash -c grep -q '^controller.quorum.voters=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 04:44:54 UTC] USER=www-data EUID=0 PID=3724178 ACTION=passthru ARGS=bash -c grep -q '^metadata.log.dir=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724197 ACTION=passthru ARGS=bash -c grep -q '^log.dirs=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724208 ACTION=fsop ARGS=sed -i s|^log.dirs=.*|log.dirs=/data/kafka/web-universe-main-dev_coordinator-data| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724217 ACTION=passthru ARGS=bash -c grep -q '^listeners=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724226 ACTION=passthru ARGS=bash -c grep -q 'CONTROLLER://' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724235 ACTION=fsop ARGS=sed -i s|^listeners=.*|listeners=SSL://10.100.1.225:9092,CONTROLLER://10.100.1.225:9093| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724254 ACTION=fsop ARGS=sed -i s|^advertised.listeners=.*|advertised.listeners=SSL://eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724263 ACTION=passthru ARGS=bash -c grep -q '^listener.security.protocol.map=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724272 ACTION=fsop ARGS=sed -i s|^listener.security.protocol.map=.*|listener.security.protocol.map=SSL:SSL,CONTROLLER:PLAINTEXT| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724283 ACTION=passthru ARGS=bash -c grep -q '^inter.broker.listener.name=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724292 ACTION=fsop ARGS=sed -i s|^inter.broker.listener.name=.*|inter.broker.listener.name=SSL| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724301 ACTION=passthru ARGS=bash -c grep -q '^offsets.topic.replication.factor=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724310 ACTION=fsop ARGS=sed -i s|^offsets.topic.replication.factor=.*|offsets.topic.replication.factor=1| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724319 ACTION=passthru ARGS=bash -c grep -q '^transaction.state.log.replication.factor=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724328 ACTION=fsop ARGS=sed -i s|^transaction.state.log.replication.factor=.*|transaction.state.log.replication.factor=1| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724350 ACTION=fsop ARGS=sed -i s|^transaction.state.log.min.isr=.*|transaction.state.log.min.isr=1| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724359 ACTION=passthru ARGS=bash -c grep -q '^min.insync.replicas=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
✅ KRaft configuration applied to server.properties
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724378 ACTION=fsop ARGS=test -f /data/kafka/web-universe-main-dev_coordinator-meta/meta.properties
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724387 ACTION=fsop ARGS=test -f /data/kafka/web-universe-main-dev_coordinator-data/meta.properties
[INFO] Already formatted: both /data/kafka/web-universe-main-dev_coordinator-meta and /data/kafka/web-universe-main-dev_coordinator-data have meta.properties
🔧 Creating/refreshing KRaft systemd unit...
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724405 ACTION=fsop ARGS=sed -i s|\\$MAINPID|$MAINPID|g /etc/systemd/system/confluent-kraft-web-universe-main-dev_coordinator.service
[2026-02-05 04:44:55 UTC] USER=www-data EUID=0 PID=3724414 ACTION=passthru ARGS=systemctl daemon-reload
✅ Ensured confluent-kraft-web-universe-main-dev_coordinator.service
🛑 Stopping legacy ZooKeeper-mode services and current KRaft instance...
🛑 Stopping current: confluent-kraft-web-universe-main-dev_coordinator.service
[2026-02-05 04:44:56 UTC] USER=www-data EUID=0 PID=3724463 ACTION=passthru ARGS=systemctl stop confluent-kraft-web-universe-main-dev_coordinator.service
🧹 Cleaning up rogue Kafka processes...
🧹 Killing any processes holding Kafka ports 9092, 9093...
🔪 Killing processes on port 9092: 3594895
3594896
3594897
3594898
3594899
3598756
[2026-02-05 04:44:59 UTC] USER=www-data EUID=0 PID=3724708 ACTION=passthru ARGS=bash -c kill -9 3594895
[2026-02-05 04:44:59 UTC] USER=www-data EUID=0 PID=3724717 ACTION=passthru ARGS=bash -c kill -9 3594896
[2026-02-05 04:44:59 UTC] USER=www-data EUID=0 PID=3724726 ACTION=passthru ARGS=bash -c kill -9 3594897
[2026-02-05 04:44:59 UTC] USER=www-data EUID=0 PID=3724737 ACTION=passthru ARGS=bash -c kill -9 3594898
[2026-02-05 04:44:59 UTC] USER=www-data EUID=0 PID=3724749 ACTION=passthru ARGS=bash -c kill -9 3594899
[2026-02-05 04:44:59 UTC] USER=www-data EUID=0 PID=3724758 ACTION=passthru ARGS=bash -c kill -9 3598756
🔪 Killing processes on port 8083: 3593645
[2026-02-05 04:45:02 UTC] USER=www-data EUID=0 PID=3725634 ACTION=passthru ARGS=bash -c kill -9 3593645
✅ Legacy services stopped and rogue processes cleaned
🔓 Removing stale lock files...
[2026-02-05 04:45:08 UTC] USER=www-data EUID=0 PID=3727894 ACTION=fsop ARGS=test -f /data/kafka/web-universe-main-dev_coordinator-data/.lock
✅ Lock file check complete
🚀 Starting confluent-kraft-web-universe-main-dev_coordinator.service ...
[2026-02-05 04:45:08 UTC] USER=www-data EUID=0 PID=3727912 ACTION=passthru ARGS=systemctl enable confluent-kraft-web-universe-main-dev_coordinator.service
[2026-02-05 04:45:09 UTC] USER=www-data EUID=0 PID=3728114 ACTION=passthru ARGS=systemctl restart confluent-kraft-web-universe-main-dev_coordinator.service
🔧 Patching shared Connect unit to follow KRaft broker...
[2026-02-05 04:45:12 UTC] USER=www-data EUID=0 PID=3730683 ACTION=fsop ARGS=sed -i -e s|${FULL_ENV}|web-universe-main-dev|g -e s|${IDENTIFIER}|coordinator|g -e s|${CONFIG_DIR}|/opt/kafka/config/web-universe-main-dev/coordinator|g /etc/systemd/system/confluent-connect-web-universe-main-dev_coordinator.service
[2026-02-05 04:45:12 UTC] USER=www-data EUID=0 PID=3730705 ACTION=fsop ARGS=sed -i s|\\$MAINPID|$MAINPID|g /etc/systemd/system/confluent-connect-web-universe-main-dev_coordinator.service
[2026-02-05 04:45:13 UTC] USER=www-data EUID=0 PID=3730800 ACTION=fsop ARGS=sed -i s|^Wants=.*|Wants=confluent-kraft-web-universe-main-dev_coordinator.service| /etc/systemd/system/confluent-connect-web-universe-main-dev_coordinator.service
[2026-02-05 04:45:13 UTC] USER=www-data EUID=0 PID=3730879 ACTION=fsop ARGS=sed -i s|^ExecStart=.*|ExecStart=/opt/kafka/bin/connect-distributed.sh /opt/kafka/config/web-universe-main-dev/coordinator/connect-distributed.properties| /etc/systemd/system/confluent-connect-web-universe-main-dev_coordinator.service
[2026-02-05 04:45:13 UTC] USER=www-data EUID=0 PID=3730904 ACTION=passthru ARGS=systemctl daemon-reload
✅ Connect unit patched
[2026-02-05 04:45:14 UTC] USER=www-data EUID=0 PID=3731111 ACTION=fsop ARGS=test -f /opt/kafka/config/web-universe-main-dev/coordinator/connect-distributed.properties
[2026-02-05 04:45:14 UTC] USER=www-data EUID=0 PID=3731141 ACTION=fsop ARGS=ln -sf /opt/kafka/config/web-universe-main-dev/coordinator/connect-distributed.properties /opt/kafka/config/connect-distributed.properties
⏳ Waiting for broker coordinator on SSL://eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092 ...
⏳ Waiting for KRaft broker... (attempt 1, 0s/600s)
Debug: Last error was: [2026-02-05 04:45:14 UTC] USER=www-data EUID=0 PID=3731155 ACTION=passthru ARGS=bash -c timeout 5 sudo -u kafka /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server 'eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092' --command-config '/etc/kafka/client-web-universe-main-dev-coordinator.properties' describe --status
⏳ Still waiting... (attempt 10, 58s/600s)
✅ coordinator responded after 117s (attempt 19)
---- server.properties (key lines) ----
[2026-02-05 04:47:39 UTC] USER=www-data EUID=0 PID=3748167 ACTION=passthru ARGS=bash -c grep -E '^(listeners|advertised\.listeners|process\.roles|controller\.quorum\.voters|controller\.listener\.names|inter\.broker\.listener\.name|log\.dirs|metadata\.log\.dir)=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
---------------------------------------
✅ KRaft setup complete for web-universe-main-dev_coordinator
server.properties : /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
data dir : /data/kafka/web-universe-main-dev_coordinator-data
meta dir : /data/kafka/web-universe-main-dev_coordinator-meta
systemd unit : confluent-kraft-web-universe-main-dev_coordinator.service
🔧 Kafka Configuration Modified:
✓ process.roles, node.id, controller.quorum.voters, controller.listener.names
✓ listeners (SSL + CONTROLLER) and advertised.listeners (FQDN fallback to IP)
✓ listener.security.protocol.map, inter.broker.listener.name
✓ log.dirs -> /data/kafka/web-universe-main-dev_coordinator-data, metadata.log.dir -> /data/kafka/web-universe-main-dev_coordinator-meta
✓ removed zookeeper.connect (if present)
✓ created/refreshed dedicated KRaft systemd unit
✓ patched shared Connect unit to follow KRaft broker
✓ symlinked /opt/kafka/config/web-universe-main-dev/coordinator/connect-distributed.properties -> /opt/kafka/config/connect-distributed.properties (compat)
🔎 Check quorum:
/opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092 --command-config /etc/kafka/client-web-universe-main-dev-coordinator.properties describe --status
📋 Next steps:
1) Review KRaft config: sudo grep -E 'process.roles|node.id|controller|listeners|advertised.listeners|log.dirs|metadata.log.dir' /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
2) Verify topics: /opt/kafka/bin/kafka-topics.sh --bootstrap-server eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092 --command-config /etc/kafka/client-web-universe-main-dev-coordinator.properties --list
✅ KRaft metadata layer setup completed successfully
Next steps:
1. Verify KRaft quorum status
2. Create Kafka topics
3. Configure Kafka Connect
[2026-02-05 04:47:39 UTC] USER=www-data EUID=0 PID=3748180 ACTION=fsop ARGS=mkdir -p /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev
[INFO] Saved metadata mode to: /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/kafka_metadata_mode
════════════════════════════════════════════════════════════════════
✅ Kafka Metadata Layer Setup Complete
Mode : kraft
Environment : web-universe-main-dev
State saved : /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/kafka_metadata_mode
KRaft cluster.id: uBayQf0-RSyd3l5SczIXzA
Verify quorum:
kafka-metadata-quorum.sh --bootstrap-server ... describe
════════════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Step 3 completed: 02-metadata.sh
[0;34m[INFO][0m 📦 Step 4/13: restart kafka related services...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 04:47:40 UTC] USER=www-data EUID=0 PID=3748244 ACTION=fsop ARGS=test -f /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 04:47:40 UTC] USER=www-data EUID=0 PID=3748254 ACTION=passthru ARGS=bash -c grep -E '^[[:space:]]*process\.roles=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties' | grep -Eq '(broker|controller)'
[INFO] 📋 Detected mode from server.properties: kraft
[2026-02-05 04:47:41 UTC] USER=www-data EUID=0 PID=3748336 ACTION=passthru ARGS=systemctl stop confluent-connect-web-universe-main-dev_coordinator.service
[2026-02-05 04:47:43 UTC] USER=www-data EUID=0 PID=3748437 ACTION=passthru ARGS=systemctl stop confluent-kafka-zk-web-universe-main-dev_coordinator.service
[2026-02-05 04:47:44 UTC] USER=www-data EUID=0 PID=3748517 ACTION=passthru ARGS=systemctl stop confluent-zookeeper-web-universe-main-dev_coordinator.service
Failed to stop confluent-zookeeper-web-universe-main-dev_coordinator.service: Unit confluent-zookeeper-web-universe-main-dev_coordinator.service not loaded.
[INFO] 🧹 Removing stale Kafka lock files...
[2026-02-05 04:47:47 UTC] USER=www-data EUID=0 PID=3748578 ACTION=fsop ARGS=rm -f /var/lib/kafka/web-universe-main-dev_coordinator-meta/.lock
[2026-02-05 04:47:47 UTC] USER=www-data EUID=0 PID=3748594 ACTION=fsop ARGS=rm -f /var/lib/kafka/web-universe-main-dev_coordinator-data/.lock
[INFO] 🧹 Cleaning up orphaned processes on Kafka ports...
[2026-02-05 04:47:47 UTC] USER=www-data EUID=0 PID=3748606 ACTION=passthru ARGS=bash -c
for port in 9092 9093 8083 2181; do
pids=$(lsof -ti tcp:$port 2>/dev/null || true)
if [[ -n "$pids" ]]; then
echo " Killing orphaned processes on port $port: $pids"
kill -9 $pids 2>/dev/null || true
sleep 1
fi
done
Killing orphaned processes on port 9092: 3728143
3731358
3731402
3731404
3731408
3739424
3739560
Killing orphaned processes on port 9093: 3594698
🚀 Restarting Kafka components…
[INFO] 🚀 starting confluent-kraft-web-universe-main-dev_coordinator.service…
[2026-02-05 04:47:51 UTC] USER=www-data EUID=0 PID=3748840 ACTION=passthru ARGS=systemctl restart confluent-kraft-web-universe-main-dev_coordinator.service
[INFO] 🚀 starting confluent-connect-web-universe-main-dev_coordinator.service…
[2026-02-05 04:47:52 UTC] USER=www-data EUID=0 PID=3749373 ACTION=passthru ARGS=systemctl restart confluent-connect-web-universe-main-dev_coordinator.service
[INFO] ⏳ Waiting for Kafka broker readiness (FQDN: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com, IP: 10.100.1.225) ...
[OK] ✅ Broker ready (attempt 1)
[OK] ✅ Port 9092 listening (Kafka Broker)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (1/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (2/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (3/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (4/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (5/40)
[OK] ✅ Port 8083 listening (Kafka Connect REST)
[INFO] ⏳ Waiting for Connect REST at https://eventbus-web-universe-main-dev-kafka-connect.fastorder.com:8083 …
[OK] ✅ Connect REST is up (attempt 1)
📋 Reconciling Connect internal topics…
[ok] connect-configs exists
[ok] connect-offsets exists
[ok] connect-status exists
═══════════════════════════════════════════════════════════════════
KAFKA SUMMARY
═══════════════════════════════════════════════════════════════════
Env: web-universe-main-dev Identifier: coordinator Mode: kraft
Broker Unit : confluent-kraft-web-universe-main-dev_coordinator.service (status: active)
Connect Unit: confluent-connect-web-universe-main-dev_coordinator.service (status: active)
Bootstrap : eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092
Connect URL : https://eventbus-web-universe-main-dev-kafka-connect.fastorder.com:8083
═══════════════════════════════════════════════════════════════════
[OK] ✅ All required services are up.
[0;32m[OK][0m ✅ Step 4 completed: 03-restart-kafka-related-services.sh
[0;34m[INFO][0m 📦 Step 5/13: checking services...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 04:49:08 UTC] USER=www-data EUID=0 PID=3755895 ACTION=fsop ARGS=test -f /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 04:49:08 UTC] USER=www-data EUID=0 PID=3755904 ACTION=passthru ARGS=bash -c grep -E '^[[:space:]]*process\.roles=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties' | grep -Eq '(broker|controller)'
[INFO] Detected mode from server.properties: kraft
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 1: Service status
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] confluent-kraft-web-universe-main-dev_coordinator.service status: active
[WARN] confluent-kafka-zk-web-universe-main-dev_coordinator.service present but should be stopped in KRaft
[WARN] confluent-zookeeper-web-universe-main-dev_coordinator.service present but not required in KRaft
[OK] confluent-connect-web-universe-main-dev_coordinator.service status: active
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 2: Port checks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Port 9092 listening (Kafka Broker)
[OK] ✅ Port 8083 listening (Kafka Connect REST)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 3: Broker readiness
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] Broker API responding (attempt 1)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 4: Kafka Connect REST
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] Connect REST responding (attempt 1)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Kafka Services Summary
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment : web-universe-main-dev
Identifier : coordinator
Mode : kraft
Broker Unit : confluent-kraft-web-universe-main-dev_coordinator.service (status: active)
Connect Unit: confluent-connect-web-universe-main-dev_coordinator.service (status: active)
Broker FQDN : eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092
Broker IP : eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092
Connect URL : https://eventbus-web-universe-main-dev-kafka-connect.fastorder.com:8083
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ All required services are reachable.
[INFO] Creating ACLs for Kafka Connect consumer groups...
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
[INFO] Creating ACLs for Connect internal topics...
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
[OK] ✅ Kafka Connect ACLs configured (deny-by-default mode)
[0;32m[OK][0m ✅ Step 5 completed: 04-checking-services.sh
[0;34m[INFO][0m 📦 Step 6/13: create audit topic...
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Creating Kafka Audit Topics
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Replication Factor: 1
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 🔍 AUDIT READINESS GATE - Preflight Checks
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m [Gate 1/5] Verifying DNS resolution...
[0;32m[OK][0m ✅ Broker DNS: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com -> 10.100.1.225
[0;32m[OK][0m ✅ Connect DNS: eventbus-web-universe-main-dev-kafka-connect.fastorder.com -> 10.100.1.75
[0;34m[INFO][0m [Gate 2/5] Verifying TLS handshake...
[0;32m[OK][0m ✅ TLS handshake: Broker certificate verified
[0;34m[INFO][0m [Gate 3/5] Verifying Kafka Connect REST API...
[0;32m[OK][0m ✅ Kafka Connect REST: Cluster ID = [2026-02-05 03:44:00 UTC] USER=www-data EUID=0 PID=3571696 ACTION=passthru ARGS=bash -c cat /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/kafka_kraft_cluster_id
uBayQf0-RSyd3l5SczIXzA
[0;34m[INFO][0m [Gate 4/5] Verifying required internal topics...
[0;32m[OK][0m ✅ Topic exists: connect-configs
[0;32m[OK][0m ✅ Topic exists: connect-offsets
[0;32m[OK][0m ✅ Topic exists: connect-status
[0;34m[INFO][0m [Gate 5/5] Verifying broker metadata access...
[0;32m[OK][0m ✅ Broker metadata: API versions accessible
[0;32m[OK][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ AUDIT READINESS GATE: ALL CHECKS PASSED
[0;32m[OK][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Waiting for Kafka to be ready...
[0;32m[OK][0m ✅ Kafka is ready
[0;34m[INFO][0m Creating audit topic: audit.events.web.universe.main.dev
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Error while executing topic command : Topic 'audit.events.web.universe.main.dev' already exists.
[2026-02-05 04:50:00,857] ERROR org.apache.kafka.common.errors.TopicExistsException: Topic 'audit.events.web.universe.main.dev' already exists.
(kafka.admin.TopicCommand$)
[0;32m[OK][0m ✅ Audit topic already exists: audit.events.web.universe.main.dev
[0;32m[OK][0m ✅ Topic verified: audit.events.web.universe.main.dev
Topic: audit.events.web.universe.main.dev TopicId: _h9pJC4FSPSnOavETTJe4Q PartitionCount: 3 ReplicationFactor: 1 Configs: compression.type=lz4,min.insync.replicas=1,cleanup.policy=delete,segment.bytes=1073741824,retention.ms=7776000000,message.timestamp.type=LogAppendTime,segment.ms=604800000
Topic: audit.events.web.universe.main.dev Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: audit.events.web.universe.main.dev Partition: 1 Leader: 1 Replicas: 1 Isr: 1
Topic: audit.events.web.universe.main.dev Partition: 2 Leader: 1 Replicas: 1 Isr: 1
[0;34m[INFO][0m Creating audit producer credentials...
Completed updating config for user audit-producer-web-universe-main-dev.
[0;32m[OK][0m ✅ Audit producer user created: audit-producer-web-universe-main-dev
[0;34m[INFO][0m Creating ACLs for audit producer...
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
[0;32m[OK][0m ✅ ACLs configured (producer: write-only, sinks: read-only, immutability: protected)
[0;34m[INFO][0m Storing audit producer credentials in AWS Secrets Manager...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/eventbus/web/universe/main/dev/kafka/audit/producer-X3Fpzs",
"Name": "fastorder/eventbus/web/universe/main/dev/kafka/audit/producer",
"VersionId": "908fc98a-a5be-49be-9db3-9a6bd11436ae"
}
[0;32m[OK][0m ✅ Credentials stored in: fastorder/eventbus/web/universe/main/dev/kafka/audit/producer
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Configuring S3 Sink for Audit Cold Storage
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
PRE audit/
[0;34m[INFO][0m S3 staging bucket exists: fastorder-audit-staging
[0;34m[INFO][0m Data flow: Kafka → fastorder-audit-staging → (replication) → fastorder-audit-immutable
[0;34m[INFO][0m Updating existing S3 sink connector...
{"name":"audit-s3-sink-web_universe_main_dev","config":{"connector.class":"io.confluent.connect.s3.S3SinkConnector","tasks.max":"1","topics":"audit.events.web.universe.main.dev","topics.dir":"audit/web-universe-main-dev","s3.bucket.name":"fastorder-audit-staging","s3.region":"me-central-1","s3.part.size":"5242880","flush.size":"1000","rotate.interval.ms":"3600000","rotate.schedule.interval.ms":"86400000","storage.class":"io.confluent.connect.s3.storage.S3Storage","format.class":"io.confluent.connect.s3.format.json.JsonFormat","partitioner.class":"io.confluent.connect.storage.partitioner.TimeBasedPartitioner","path.format":"'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH","partition.duration.ms":"3600000","locale":"en-US","timezone":"UTC","timestamp.extractor":"Record","key.converter":"org.apache.kafka.connect.json.JsonConverter","value.converter":"org.apache.kafka.connect.json.JsonConverter","key.converter.schemas.enable":"false","value.converter.schemas.enable":"false","behavior.on.null.values":"ignore","errors.tolerance":"all","errors.log.enable":"true","errors.log.include.messages":"true","name":"audit-s3-sink-web_universe_main_dev"},"tasks":[{"connector":"audit-s3-sink-web_universe_main_dev","task":0}],"type":"sink"}[0;32m[OK][0m ✅ S3 Sink connector configured for audit cold storage
[0;34m[INFO][0m Staging Bucket: fastorder-audit-staging (Kafka Connect writes here)
[0;34m[INFO][0m Immutable Bucket: fastorder-audit-immutable (via S3 Replication)
[0;34m[INFO][0m Path: audit/web-universe-main-dev/
[0;34m[INFO][0m Final Retention: WORM-enabled (Object Lock COMPLIANCE mode, 1-year)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Kafka Audit Topic Created Successfully
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Topic: audit.events.web.universe.main.dev
[0;34m[INFO][0m Partitions: 3
[0;34m[INFO][0m Replication Factor: 1
[0;34m[INFO][0m Retention: 90 days
[0;34m[INFO][0m Producer: audit-producer-web-universe-main-dev (write-only)
[0;34m[INFO][0m Application Integration:
[0;34m[INFO][0m - Use credentials from: fastorder/eventbus/web/universe/main/dev/kafka/audit/producer
[0;34m[INFO][0m - Connect to: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092
[0;34m[INFO][0m - Produce to: audit.events.web.universe.main.dev
[0;34m[INFO][0m - Security: SASL_SSL (SCRAM-SHA-512)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 📋 PCI-DSS Compliance Status
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m
[0;34m[INFO][0m ✅ Kafka Hot Storage: 90 days (meets PCI-DSS 3-month immediate access)
[0;34m[INFO][0m ✅ ACL Authorization: deny-by-default (allow.everyone.if.no.acl.found=false)
[0;34m[INFO][0m ✅ Immutability: DENY DELETE/ALTER on audit topic
[0;34m[INFO][0m ✅ S3 Cold Storage: fastorder-audit-immutable (Object Lock COMPLIANCE, 1-year)
[0;34m[INFO][0m
[0;34m[INFO][0m S3 Audit Storage:
[0;34m[INFO][0m Bucket: s3://fastorder-audit-staging
[0;34m[INFO][0m Path: audit/web-universe-main-dev/
[0;34m[INFO][0m Object Lock: COMPLIANCE mode, 1-year retention
[0;34m[INFO][0m Immutability: Objects cannot be deleted or modified for 1 year
[0;34m[INFO][0m
[0;34m[INFO][0m Verify compliance with:
[0;34m[INFO][0m bash 04-eventbus/engine/kafka/steps/11-audit-compliance-check.sh
[0;34m[INFO][0m
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 📋 AWS Roles Anywhere - Credential Refresh Setup
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m
[0;34m[INFO][0m For S3 sink to write to S3, Kafka Connect needs AWS credentials.
[0;34m[INFO][0m Use IAM Roles Anywhere with systemd timer for automatic refresh.
[0;34m[INFO][0m
[0;34m[INFO][0m Required files:
[0;34m[INFO][0m Certificate: /etc/fastorder/rolesanywhere/client-bundle.crt
[0;34m[INFO][0m Private Key: /etc/fastorder/rolesanywhere/client.key
[0;34m[INFO][0m Helper: /usr/local/bin/aws_signing_helper
[0;34m[INFO][0m
[0;34m[INFO][0m Systemd timer: kafka-aws-credential-refresh.timer
[0;34m[INFO][0m Runs every 30 minutes to refresh credentials to /var/lib/kafka/.aws/credentials
[0;34m[INFO][0m
[0;34m[INFO][0m Verify timer is active:
[0;34m[INFO][0m systemctl status kafka-aws-credential-refresh.timer
[0;34m[INFO][0m
[0;34m[INFO][0m Documentation: https://docs.aws.amazon.com/rolesanywhere/latest/userguide/
[0;32m[OK][0m ✅ Step 6 completed: 05-create-audit-topic.sh
[0;34m[INFO][0m 📦 Step 7/13: setup backups...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Kafka Backup Configuration
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Environment: web-universe-main-dev
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[0;34m[INFO][0m 1️⃣ Creating S3 bucket for Kafka backups...
make_bucket failed: s3://fastorder-kafka-backups-web-universe-main-dev An error occurred (AccessDenied) when calling the CreateBucket operation: User: arn:aws:iam::464621692046:user/fo-dev is not authorized to perform: s3:CreateBucket on resource: "arn:aws:s3:::fastorder-kafka-backups-web-universe-main-dev" because no identity-based policy allows the s3:CreateBucket action
An error occurred (NoSuchBucket) when calling the PutBucketVersioning operation: The specified bucket does not exist
Parameter validation failed:
Unknown parameter in LifecycleConfiguration.Rules[0]: "Id", must be one of: Expiration, ID, Prefix, Filter, Status, Transitions, NoncurrentVersionTransitions, NoncurrentVersionExpiration, AbortIncompleteMultipartUpload
[0;32m[OK][0m ✅ S3 bucket created: fastorder-kafka-backups-web-universe-main-dev
[0;34m[INFO][0m 2️⃣ Creating local backup directory...
[2026-02-05 04:50:49 UTC] USER=www-data EUID=0 PID=3760828 ACTION=fsop ARGS=mkdir -p /var/backups/kafka/web-universe-main-dev
[2026-02-05 04:50:49 UTC] USER=www-data EUID=0 PID=3760840 ACTION=fsop ARGS=mkdir -p /var/backups/kafka/web-universe-main-dev/topics
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3760849 ACTION=fsop ARGS=mkdir -p /var/backups/kafka/web-universe-main-dev/metadata
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3760864 ACTION=fsop ARGS=mkdir -p /var/log/kafka/backups
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3760873 ACTION=fsop ARGS=chown -R kafka:kafka /var/backups/kafka/web-universe-main-dev
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3760882 ACTION=fsop ARGS=chown -R kafka:kafka /var/log/kafka/backups
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3760891 ACTION=fsop ARGS=chmod 750 /var/backups/kafka/web-universe-main-dev
[0;32m[OK][0m ✅ Local backup directory created
[0;34m[INFO][0m 3️⃣ Creating topic backup script...
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3760909 ACTION=fsop ARGS=sed -i s|__ENV_ID__|web-universe-main-dev|g /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3760918 ACTION=fsop ARGS=sed -i s|__KAFKA_BROKER__|eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092|g /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3760927 ACTION=fsop ARGS=sed -i s|__BACKUP_DIR__|/var/backups/kafka/web-universe-main-dev|g /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3760936 ACTION=fsop ARGS=sed -i s|__S3_BUCKET__|fastorder-kafka-backups-web-universe-main-dev|g /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3760963 ACTION=fsop ARGS=chmod 750 /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3760978 ACTION=fsop ARGS=chown root:kafka /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[0;32m[OK][0m ✅ Backup script created: /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[0;34m[INFO][0m 4️⃣ Setting up cron jobs for automated backups...
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3761017 ACTION=fsop ARGS=chmod 644 /etc/cron.d/kafka-backups-web-universe-main-dev
[0;32m[OK][0m ✅ Cron job configured: Daily backups at 2:00 AM
[0;34m[INFO][0m 5️⃣ Creating restore documentation...
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3761046 ACTION=fsop ARGS=sed -i s|__S3_BUCKET__|fastorder-kafka-backups-web-universe-main-dev|g /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3761058 ACTION=fsop ARGS=sed -i s|__S3_REGION__|me-central-1|g /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3761067 ACTION=fsop ARGS=sed -i s|__KAFKA_BROKER__|eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com|g /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3761076 ACTION=fsop ARGS=chmod 644 /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3761085 ACTION=fsop ARGS=chown kafka:kafka /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[0;32m[OK][0m ✅ Restore documentation created: /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Kafka Backup Configured
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m S3 Bucket: fastorder-kafka-backups-web-universe-main-dev
[0;34m[INFO][0m Region: me-central-1
[0;34m[INFO][0m Local backup dir: /var/backups/kafka/web-universe-main-dev
[0;34m[INFO][0m Schedule: Daily at 2:00 AM
[0;34m[INFO][0m Script: /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[0;34m[INFO][0m Restore docs: /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[1;33m[WARN][0m ⚠️ Note: This backs up Kafka metadata only (topics, configs, offsets)
[1;33m[WARN][0m For full message data backup, configure Kafka Connect S3 Sink
[0;32m[OK][0m ✅ Step 7 completed: 06-setup-backups.sh
[0;34m[INFO][0m 📦 Step 8/13: monitoring setup...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Kafka Monitoring Integration for web-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-web-universe-main-dev
[1;32m[OK][0m Observability cell endpoints registered for web-universe-main-dev
[1;32m[OK][0m ✓ Observability cell is ready
[INFO] 2️⃣ Setting up Kafka JMX exporter integration...
[INFO] JMX Exporter port calculated for web-universe-main-dev: 9362 (offset: 54)
[INFO] Checking observability cell readiness: obs-web-universe-main-dev
[1;32m[OK][0m Observability cell endpoints registered for web-universe-main-dev
[INFO] Setting up Kafka JMX exporter for web-universe-main-dev
[INFO] JMX Prometheus Java Agent already exists at /opt/kafka/libs/jmx_prometheus_javaagent.jar
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3761133 ACTION=passthru ARGS=mv /tmp/jmx_exporter.yml /opt/kafka/config/jmx_exporter.yml
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3761142 ACTION=passthru ARGS=chmod 644 /opt/kafka/config/jmx_exporter.yml
[1;32m[OK][0m JMX exporter configuration created at /opt/kafka/config/jmx_exporter.yml
[1;32m[OK][0m JMX exporter configuration created
[INFO] Configuring Kafka systemd services to use JMX exporter...
[2026-02-05 04:50:50 UTC] USER=www-data EUID=0 PID=3761169 ACTION=fsop ARGS=test -f /etc/systemd/system/[2026-02-05
[INFO] All Kafka services already configured with JMX exporter
[1;32m[OK][0m Kafka JMX exporter integration complete
[INFO] Metrics endpoint: http://142.93.238.16:9362/metrics
[INFO] Prometheus will automatically scrape: https://metrics-web-universe-main-dev.fastorder.com:9090
[INFO] View dashboards at: https://dashboards-web-universe-main-dev.fastorder.com
[1;32m[OK][0m ✓ Kafka JMX exporter integration complete
[INFO] Configuring KAFKA_OPTS environment variable for kafka user...
[2026-02-05 04:50:51 UTC] USER=www-data EUID=0 PID=3761211 ACTION=passthru ARGS=sed -i /export KAFKA_OPTS=.*jmx_prometheus_javaagent/d /home/kafka/.bashrc
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[1;32m[OK][0m ✓ KAFKA_OPTS configured in /home/kafka/.bashrc (port 9362)
[INFO] 2.5️⃣ Enabling JMX exporter in Kafka systemd service...
[2026-02-05 04:50:51 UTC] USER=www-data EUID=0 PID=3761248 ACTION=passthru ARGS=grep -q javaagent.*jmx_prometheus_javaagent /etc/systemd/system/confluent-kraft-web-universe-main-dev_coordinator.service
[INFO] Updating confluent-kraft-web-universe-main-dev_coordinator.service to enable JMX exporter...
[2026-02-05 04:50:51 UTC] USER=www-data EUID=0 PID=3761283 ACTION=passthru ARGS=sed -i s|^Environment=KAFKA_OPTS=.*|Environment=KAFKA_OPTS=-javaagent:/opt/kafka/libs/jmx_prometheus_javaagent.jar=9362:/opt/kafka/config/jmx_exporter.yml| /etc/systemd/system/confluent-kraft-web-universe-main-dev_coordinator.service
[1;32m[OK][0m ✓ Updated confluent-kraft-web-universe-main-dev_coordinator.service
[INFO] Reloading systemd daemon and restarting Kafka services...
[2026-02-05 04:50:51 UTC] USER=www-data EUID=0 PID=3761306 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 04:50:52 UTC] USER=www-data EUID=0 PID=3761370 ACTION=passthru ARGS=systemctl is-active --quiet confluent-kraft-web-universe-main-dev_coordinator
[INFO] Restarting confluent-kraft-web-universe-main-dev_coordinator...
[2026-02-05 04:50:52 UTC] USER=www-data EUID=0 PID=3761391 ACTION=passthru ARGS=systemctl restart confluent-kraft-web-universe-main-dev_coordinator
[2026-02-05 04:50:56 UTC] USER=www-data EUID=0 PID=3762071 ACTION=passthru ARGS=systemctl is-active --quiet confluent-kraft-web-universe-main-dev_coordinator
[1;32m[OK][0m ✓ confluent-kraft-web-universe-main-dev_coordinator restarted successfully
[1;32m[OK][0m ✓ JMX exporter enabled in Kafka systemd services
[INFO] 2.6️⃣ Configuring Prometheus to scrape Kafka metrics...
[2026-02-05 04:50:56 UTC] USER=www-data EUID=0 PID=3762095 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
[INFO] Adding Kafka scrape target to Prometheus configuration...
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[2026-02-05 04:50:56 UTC] USER=www-data EUID=0 PID=3762139 ACTION=passthru ARGS=sed -i /# Prometheus self-monitoring/r /tmp/prometheus_kafka_add.yml /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
[ERROR] Invalid Prometheus configuration - rolling back
[2026-02-05 04:50:57 UTC] USER=www-data EUID=0 PID=3762173 ACTION=passthru ARGS=sed -i /job_name: 'kafka'/,+6d /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
[2026-02-05 04:50:57 UTC] USER=www-data EUID=0 PID=3762194 ACTION=fsop ARGS=rm -f /tmp/prometheus_kafka_add.yml
[INFO] 3️⃣ Registering Kafka nodes to monitoring database...
[INFO] Detected Kafka version: 3.9.1
[INFO] Registering Kafka Broker to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Kafka Broker
[INFO] Identifier: web-universe-main-dev-broker-01
[INFO] Identifier Parent: cluster
[INFO] IP: 142.93.238.16
[INFO] Port: 9092
[INFO] FQDN: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 0fe22eef-a876-408e-9099-f79ee8d192b7
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[1;32m[OK][0m ✓ Kafka broker registered
[INFO] Registering Kafka Connect to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Kafka Connect
[INFO] Identifier: web-universe-main-dev-connect-01
[INFO] Identifier Parent: cluster
[INFO] IP: 142.93.238.16
[INFO] Port: 8083
[INFO] FQDN: eventbus-web-universe-main-dev-kafka-connect.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 269c6f12-e045-4268-8bc4-73c5e936d212
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[1;32m[OK][0m ✓ Kafka Connect registered
[INFO] Schema Registry not running, skipping registration
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Kafka Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Metrics: http://localhost:9362/metrics
[INFO] Prometheus: https://metrics-web-universe-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-web-universe-main-dev.fastorder.com
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Step 8 completed: 10-monitoring-setup.sh
[0;34m[INFO][0m 📦 Step 9/13: audit compliance check...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[1m KAFKA AUDIT COMPLIANCE DASHBOARD - PCI-DSS Verification[0m
[1m Environment: web-universe-main-dev[0m
[1m Timestamp: 2026-02-05 04:51:01 UTC[0m
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[1m[1/5] Kafka Deny-by-Default ACL Posture[0m
Requirement: allow.everyone.if.no.acl.found=false
[0;32mPASS[0m Deny-by-default is ENABLED (allow.everyone.if.no.acl.found=false)
[1m[2/5] Audit Topic Hot Retention (90 days)[0m
Requirement: retention.ms >= 7776000000 (90 days)
[0;32mPASS[0m Retention is 90 days (7776000000 ms)
[1m[3/5] Kafka Connect S3 Sink Status[0m
Requirement: Connector and all tasks RUNNING
[0;31mFAIL[0m Connector RUNNING but tasks FAILED: FAILED
[1m[4/5] S3 Freshness Evidence[0m
Requirement: Newest object < 120 minutes old
[1;33mWARN[0m No objects found in s3://fastorder-audit-immutable/audit/web-universe-main-dev/
This may be normal if no audit events have been generated yet
[1m[5/5] S3 Object Lock Immutability[0m
Requirement: COMPLIANCE mode with 1-year retention
[0;31mFAIL[0m Cannot verify Object Lock configuration - access denied
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[1m COMPLIANCE SUMMARY[0m
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;31m[1mCOMPLIANCE ISSUES DETECTED[0m
Passed: 2/5
Failed: 3/5
Review failed checks above and remediate.
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;32m[OK][0m ✅ Step 9 completed: 11-audit-compliance-check.sh
[0;34m[INFO][0m 📦 Step 10/13: audit canary test...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[1m KAFKA AUDIT CANARY TEST - End-to-End Verification[0m
[1m Environment: web-universe-main-dev[0m
[1m Canary ID: canary-1770267073-3763637[0m
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[1m[Step 1/4] Producing canary audit event to Kafka[0m
Topic: audit.events.web.universe.main.dev
Event: canary-1770267073-3763637
[0;31mFailed to produce event[0m
[1;33m(Topic may not exist yet - normal during initial setup)[0m
[0;32m[OK][0m ✅ Step 10 completed: 12-audit-canary-test.sh
[0;34m[INFO][0m 📦 Step 11/13: setup audit s3 staging...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
════════════════════════════════════════════════════════════════════════
Kafka Audit S3 Staging + Replication Setup
════════════════════════════════════════════════════════════════════════
Staging Bucket: fastorder-audit-staging
Immutable Bucket: fastorder-audit-immutable
Region: me-central-1
Environment: --auto
════════════════════════════════════════════════════════════════════════
[INFO] 1️⃣ Checking AWS credentials...
[WARN] No AWS credentials found - skipping S3 staging setup
To configure S3 audit storage later, add credentials to /home/ab/.aws/credentials:
[admin]
aws_access_key_id = AKIA...
aws_secret_access_key = ...
Then run: AWS_PROFILE=admin /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps/13-setup-audit-s3-staging.sh --auto
[0;32m[OK][0m ✅ Step 11 completed: 13-setup-audit-s3-staging.sh
[0;34m[INFO][0m 📦 Step 12/13: install ksqldb...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ksqlDB Installation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Identifier: --auto
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Allocating new VM_IP for ksqlDB: 10.100.1.242
🔧 Adding VM_IP 10.100.1.242 to loopback interface...
[2026-02-05 04:51:19 UTC] USER=www-data EUID=0 PID=3764290 ACTION=configure-network-interface ARGS=lo 10.100.1.242
✓ lo <- 10.100.1.242
[2026-02-05 04:51:19 UTC] USER=www-data EUID=0 PID=3764326 ACTION=fsop ARGS=tee -a /etc/hosts
10.100.1.242 eventbus-web-universe-main-dev-ksqldb---auto.fastorder.com
VM_IP: 10.100.1.242
FQDN: eventbus-web-universe-main-dev-ksqldb---auto.fastorder.com
📦 Step 1: Checking Confluent Platform installation...
✅ ksqlDB already installed (version: )
📁 Step 2: Creating directories...
[2026-02-05 04:51:23 UTC] USER=www-data EUID=0 PID=3764464 ACTION=fsop ARGS=mkdir -p /var/lib/ksqldb/web-universe-main-dev/--auto
[2026-02-05 04:51:23 UTC] USER=www-data EUID=0 PID=3764485 ACTION=fsop ARGS=mkdir -p /var/log/ksqldb/web-universe-main-dev/--auto
[2026-02-05 04:51:23 UTC] USER=www-data EUID=0 PID=3764529 ACTION=fsop ARGS=chown -R kafka:kafka /var/lib/ksqldb/web-universe-main-dev/--auto /var/log/ksqldb/web-universe-main-dev/--auto /etc/ksqldb/web-universe-main-dev/--auto
✅ Directories created
⚙️ Step 3: Generating ksqlDB configuration...
[2026-02-05 04:51:23 UTC] USER=www-data EUID=0 PID=3764552 ACTION=fsop ARGS=mv /tmp/ksql-server-web-universe-main-dev.properties /etc/ksqldb/web-universe-main-dev/--auto/ksql-server.properties
[2026-02-05 04:51:23 UTC] USER=www-data EUID=0 PID=3764573 ACTION=fsop ARGS=chown kafka:kafka /etc/ksqldb/web-universe-main-dev/--auto/ksql-server.properties
[2026-02-05 04:51:23 UTC] USER=www-data EUID=0 PID=3764595 ACTION=fsop ARGS=chmod 640 /etc/ksqldb/web-universe-main-dev/--auto/ksql-server.properties
✅ Configuration generated: /etc/ksqldb/web-universe-main-dev/--auto/ksql-server.properties
🔧 Step 4: Creating systemd service...
[2026-02-05 04:51:23 UTC] USER=www-data EUID=0 PID=3764618 ACTION=fsop ARGS=mv /tmp/ksqldb-web-universe-main-dev---auto.service /etc/systemd/system/ksqldb-web-universe-main-dev---auto.service
[2026-02-05 04:51:23 UTC] USER=www-data EUID=0 PID=3764639 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 04:51:24 UTC] USER=www-data EUID=0 PID=3764698 ACTION=passthru ARGS=systemctl enable ksqldb-web-universe-main-dev---auto.service
Created symlink /etc/systemd/system/multi-user.target.wants/ksqldb-web-universe-main-dev---auto.service → /etc/systemd/system/ksqldb-web-universe-main-dev---auto.service.
✅ Systemd service created: ksqldb-web-universe-main-dev---auto.service
🚀 Step 5: Starting ksqlDB service...
🔍 Checking Kafka broker connectivity...
✅ Kafka broker is accessible
[2026-02-05 04:51:25 UTC] USER=www-data EUID=0 PID=3764774 ACTION=passthru ARGS=systemctl start ksqldb-web-universe-main-dev---auto.service
✅ ksqlDB service started
⏳ Waiting for ksqlDB to be ready...
..............................
🔍 Step 6: Verifying installation...
📊 Service Status:
[2026-02-05 04:52:25 UTC] USER=www-data EUID=0 PID=3766834 ACTION=passthru ARGS=systemctl status ksqldb-web-universe-main-dev---auto.service --no-pager -l
● ksqldb-web-universe-main-dev---auto.service - ksqlDB Server (web-universe-main-dev --auto)
Loaded: loaded (/etc/systemd/system/ksqldb-web-universe-main-dev---auto.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2026-02-05 04:52:23 UTC; 2s ago
Docs: https://docs.ksqldb.io/
Main PID: 3766676 (java)
Tasks: 24 (limit: 19051)
Memory: 219.1M
CPU: 4.151s
CGroup: /system.slice/ksqldb-web-universe-main-dev---auto.service
└─3766676 java -cp "/usr/share/java/ksqldb/*:/usr/share/java/rest-utils/*:/usr/share/java/confluent-common/*:" -Xms256m -Xmx512m -server -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:NewRatio=1 -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dksql.log.dir=/var/log/ksqldb/web-universe-main-dev/--auto -Dlog4j.configuration=file:/etc/ksqldb/log4j.properties -Dksql.server.install.dir=/usr "-Xlog:gc*:file=/var/log/ksqldb/web-universe-main-dev/--auto/ksql-server-gc.log:time,tags:filecount=10,filesize=102400" io.confluent.ksql.rest.server.KsqlServerMain /etc/ksqldb/web-universe-main-dev/--auto/ksql-server.properties
Feb 05 04:52:25 web-03 ksql-server-start[3766676]: [2026-02-05 04:52:25,814] INFO Adding function cos for method public java.lang.Double io.confluent.ksql.function.udf.math.Cos.cos(java.lang.Double) (io.confluent.ksql.function.UdfLoader:147)
Feb 05 04:52:25 web-03 ksql-server-start[3766676]: [2026-02-05 04:52:25,814] INFO Adding function cos for method public java.lang.Double io.confluent.ksql.function.udf.math.Cos.cos(java.lang.Long) (io.confluent.ksql.function.UdfLoader:147)
Feb 05 04:52:25 web-03 ksql-server-start[3766676]: [2026-02-05 04:52:25,849] INFO Adding function cos for method public java.lang.Double io.confluent.ksql.function.udf.math.Cos.cos(java.lang.Integer) (io.confluent.ksql.function.UdfLoader:147)
📊 ksqlDB Info:
⚠️ ksqlDB not responding yet (may still be starting)
📡 Step 7: Registering ksqlDB to Observability API...
🔄 Registering ksqlDB node to observability dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: ksqlDB
[INFO] Identifier: web-universe-main-dev-ksqldb---auto
[INFO] Identifier Parent: eventbus
[INFO] IP: 10.100.1.242
[INFO] Port: 8088
[INFO] FQDN: eventbus-web-universe-main-dev-ksqldb---auto.fastorder.com
[INFO] Status: starting
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 959bfeea-5527-4a0f-84cb-9c8e8a9d7811
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
✅ ksqlDB registered successfully
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ksqlDB Installation Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Service: ksqldb-web-universe-main-dev---auto
VM_IP: 10.100.1.242
FQDN: eventbus-web-universe-main-dev-ksqldb---auto.fastorder.com
Port: 8088
Config: /etc/ksqldb/web-universe-main-dev/--auto/ksql-server.properties
Data: /var/lib/ksqldb/web-universe-main-dev/--auto
Logs: /var/log/ksqldb/web-universe-main-dev/--auto
Dashboard:
https://skeleton.dev.fastorder.com/dashboard/monitoring/environment2/<env-id>/service/ksqldb
CLI Access (with SSL):
ksql --ssl https://eventbus-web-universe-main-dev-ksqldb---auto.fastorder.com:8088
REST API (HTTPS):
curl -k https://eventbus-web-universe-main-dev-ksqldb---auto.fastorder.com:8088/info
curl -k https://eventbus-web-universe-main-dev-ksqldb---auto.fastorder.com:8088/ksql -H 'Content-Type: application/vnd.ksql.v1+json' -d '{"ksql": "SHOW STREAMS;"}'
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Step 12 completed: 20-install-ksqldb.sh
[0;34m[INFO][0m 📦 Step 13/13: update www data certs...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
📋 Post-Kafka Setup: Updating www-data Kafka certificates...
Environment: web-universe-main-dev
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
✓ Kafka certificates found
✓ www-data user exists
[2026-02-05 04:52:26 UTC] USER=www-data EUID=0 PID=3766950 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:52:26 UTC] USER=www-data EUID=0 PID=3766961 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:52:27 UTC] USER=www-data EUID=0 PID=3766970 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
✅ Symlinked client-cert.pem
[2026-02-05 04:52:27 UTC] USER=www-data EUID=0 PID=3766988 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 04:52:27 UTC] USER=www-data EUID=0 PID=3766997 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:52:27 UTC] USER=www-data EUID=0 PID=3767006 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:52:27 UTC] USER=www-data EUID=0 PID=3767015 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:52:27 UTC] USER=www-data EUID=0 PID=3767024 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks created for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✓ Post-Kafka setup complete
[0;32m[OK][0m ✅ Step 13 completed: 99-update-www-data-certs.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Kafka setup completed successfully!
[0;32m[OK][0m Executed all 13 steps
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Service: web
[0;34m[INFO][0m Zone: universe
[0;34m[INFO][0m Branch: main
[0;34m[INFO][0m Env: dev
[0;34m[INFO][0m Registering Kafka nodes via API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Kafka
[INFO] Identifier: web-universe-main-dev_coordinator
[INFO] Identifier Parent: coordinator
[INFO] IP: 10.100.1.225
[INFO] Port: 9092
[INFO] FQDN: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 556513cd-577f-4835-837d-7f8a97c24183
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Kafka Connect
[INFO] Identifier: web-universe-main-dev_coordinator
[INFO] Identifier Parent: coordinator
[INFO] IP: 10.100.1.75
[INFO] Port: 8083
[INFO] FQDN: eventbus-web-universe-main-dev-kafka-connect.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 832e2b3d-94f8-4caf-9464-57bb9914f0a8
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[0;32m[OK][0m ✔ Kafka node registration completed
[0;34m[INFO][0m Setting up Kafka observability integration...
[0;34m[INFO][0m Checking observability cell readiness: obs-web-universe-main-dev
[0;32m[OK][0m Observability cell endpoints registered for web-universe-main-dev
[0;34m[INFO][0m Observability cell verified for web-universe-main-dev
[0;34m[INFO][0m Monitoring will be configured after Kafka deployment (step 10-monitoring-setup.sh)
[0;34m[INFO][0m Cleaning up temporary files...
[INFO] Starting cleanup of temporary files...
[INFO] Cleaning up SSL temp files for web-universe-main-dev...
[INFO] Cleaning up old provisioning logs...
[INFO] Cleaning up old configuration backups...
[0;32m[OK][0m ✔ Cleanup completed
[0;32m✓[0m ✅ Event bus infrastructure (kafka) setup completed successfully
[0;34m[INFO][0m Using database engine from DB_ENGINE environment variable: postgresql
[0;34m[INFO][0m Cleaning up any existing locks...
[0;32m[1mStarting database engine: postgresql[0m
[1;33m═══════════════════════════════════════════════[0m
[0;34m[INFO][0m Using environment from web interface: web-universe-main-dev
[0;32m[2026-02-05 04:52:28][0m Using web-provided environment: web-universe-main-dev
[0;32m[2026-02-05 04:52:28][0m Service: web, Zone: universe, Branch: main, Env: dev
[0;32m✓[0m Environment initialized successfully (mode: general)
[0;34m[INFO][0m Checking observability cell readiness: obs-web-universe-main-dev
[1;32m[OK][0m Observability cell endpoints registered for web-universe-main-dev
[0;34m[INFO][0m Observability cell verified for web-universe-main-dev
[0;34m[INFO][0m Monitoring will be configured after PostgreSQL deployment (step 10-monitoring-setup.sh)
[0;34m[INFO][0m Citus mode ENABLED
[0;34m[INFO][0m → Coordinator + 1 worker(s) + 0 standby node(s) per worker
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up coordinator (Citus control plane)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-02-05 04:52:29 UTC] USER=unknown EUID=33 PID=3767331 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 04:52:29 UTC] USER=unknown EUID=33 PID=3767338 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 04:52:29 UTC] USER=unknown EUID=33 PID=3767346 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-02-05 04:52:29 UTC] USER=unknown EUID=33 PID=3767353 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 04:52:29 UTC] USER=unknown EUID=33 PID=3767360 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-02-05 04:52:29 UTC] USER=unknown EUID=33 PID=3767367 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for web in universe-dev...
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=752a811e-1a02-4989-b8c6-1afd6a8f211b JOB_UUID=1afa4e23-6aff-4492-82ab-adcbd43dadaa
[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=752a811e-1a02-4989-b8c6-1afd6a8f211b)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[WARN] Could not find PostgreSQL IP for coordinator in topology.json, allocating new VM IP...
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Allocated new VM IP: 10.100.1.119 for db-coordinator-postgresql
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: coordinator
[INFO] PostgreSQL IP: 10.100.1.119
[INFO] Primary hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[INFO] Adding /etc/hosts entries for coordinator...
[INFO] 1. db-web-universe-main-dev-postgresql.fastorder.com → 10.100.1.119 (primary/short)
[INFO] 2. db-web-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.119 (compatibility)
[INFO] ➕ Adding db-web-universe-main-dev-postgresql.fastorder.com → 10.100.1.119
[0;32m✅[0m ✅ Added: db-web-universe-main-dev-postgresql.fastorder.com → 10.100.1.119
[INFO] ➕ Adding db-web-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.119
[0;32m✅[0m ✅ Added: db-web-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.119
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
10.100.1.119 db-web-universe-main-dev-postgresql.fastorder.com
10.100.1.119 db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=752a811e-1a02-4989-b8c6-1afd6a8f211b)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe (FastOrder Universe)
Branch: main
Env: dev
Node: coordinator
Primary CN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Alt CN: web-universe-main-dev.fastorder.com
VM IP: 142.93.238.16
Coordinator variants:
- db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
- db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 04:52:33 UTC] USER=www-data EUID=0 PID=3767748 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator and /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:52:33 UTC] USER=www-data EUID=0 PID=3767757 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-02-05 04:52:33 UTC] USER=www-data EUID=0 PID=3767767 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-3767715
[2026-02-05 04:52:33 UTC] USER=www-data EUID=0 PID=3767776 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-3767715/ra_root.crt
[2026-02-05 04:52:33 UTC] USER=www-data EUID=0 PID=3767785 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-3767715/ra_root.key
[2026-02-05 04:52:33 UTC] USER=www-data EUID=0 PID=3767794 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3767715/ra_root.crt
[2026-02-05 04:52:33 UTC] USER=www-data EUID=0 PID=3767803 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3767715/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[2026-02-05 04:52:36 UTC] USER=www-data EUID=0 PID=3767897 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3767715/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 04:52:36 UTC] USER=www-data EUID=0 PID=3767918 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3767715/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 04:52:37 UTC] USER=www-data EUID=0 PID=3767927 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-02-05 04:52:37 UTC] USER=www-data EUID=0 PID=3767937 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3767715/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:52:37 UTC] USER=www-data EUID=0 PID=3767949 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:52:37 UTC] USER=www-data EUID=0 PID=3767969 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:52:37 UTC] USER=www-data EUID=0 PID=3767987 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 04:52:37 UTC] USER=www-data EUID=0 PID=3768001 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 04:52:37 UTC] USER=www-data EUID=0 PID=3768027 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 04:52:37 UTC] USER=www-data EUID=0 PID=3768036 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 04:52:37 UTC] USER=www-data EUID=0 PID=3768045 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 04:52:37 UTC] USER=www-data EUID=0 PID=3768055 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:52:37 UTC] USER=www-data EUID=0 PID=3768066 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔍 Verifying certificate...
Certificate details:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Node: coordinator
Primary CN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-coordinator.service
3. Test SSL connection:
psql "host=db-web-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: postgres
Identifier: coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: coordinator
User (CN): postgres
Hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 04:52:38 UTC] USER=www-data EUID=0 PID=3768161 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-02-05 04:52:38 UTC] USER=www-data EUID=0 PID=3768173 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 04:52:39 UTC] USER=www-data EUID=0 PID=3768195 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 04:52:39 UTC] USER=www-data EUID=0 PID=3768204 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:52:39 UTC] USER=www-data EUID=0 PID=3768243 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:52:39 UTC] USER=www-data EUID=0 PID=3768253 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:52:39 UTC] USER=www-data EUID=0 PID=3768265 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 04:52:40 UTC] USER=www-data EUID=0 PID=3768288 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 04:52:40 UTC] USER=www-data EUID=0 PID=3768321 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:52:40 UTC] USER=www-data EUID=0 PID=3768332 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 04:52:40 UTC] USER=www-data EUID=0 PID=3768343 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 04:52:40 UTC] USER=www-data EUID=0 PID=3768352 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 04:52:40 UTC] USER=www-data EUID=0 PID=3768361 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 04:52:40 UTC] USER=www-data EUID=0 PID=3768372 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:52:40 UTC] USER=www-data EUID=0 PID=3768381 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:52:40 UTC] USER=www-data EUID=0 PID=3768390 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 04:52:40 UTC] USER=www-data EUID=0 PID=3768399 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 04:52:40 UTC] USER=www-data EUID=0 PID=3768408 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 04:52:40 UTC] USER=www-data EUID=0 PID=3768434 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:52:40 UTC] USER=www-data EUID=0 PID=3768445 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:41 UTC] USER=www-data EUID=0 PID=3768473 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:41 UTC] USER=www-data EUID=0 PID=3768482 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:52:41 UTC] USER=www-data EUID=0 PID=3768491 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:52:41 UTC] USER=www-data EUID=0 PID=3768500 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:41 UTC] USER=www-data EUID=0 PID=3768509 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:41 UTC] USER=www-data EUID=0 PID=3768518 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 04:52:41 UTC] USER=www-data EUID=0 PID=3768530 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 04:52:41 UTC] USER=www-data EUID=0 PID=3768539 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:52:41 UTC] USER=www-data EUID=0 PID=3768575 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 04:52:41 UTC] USER=www-data EUID=0 PID=3768585 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:41 UTC] USER=www-data EUID=0 PID=3768596 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768608 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768619 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768634 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768643 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768652 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768661 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768680 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768689 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768698 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768707 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768716 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768726 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768736 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768745 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768754 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768767 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768777 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768787 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768796 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768805 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768814 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768823 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768832 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768841 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768851 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:42 UTC] USER=www-data EUID=0 PID=3768862 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:43 UTC] USER=www-data EUID=0 PID=3768871 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:52:43 UTC] USER=www-data EUID=0 PID=3768880 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:52:43 UTC] USER=www-data EUID=0 PID=3768889 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:43 UTC] USER=www-data EUID=0 PID=3768898 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:43 UTC] USER=www-data EUID=0 PID=3768907 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 04:52:43 UTC] USER=www-data EUID=0 PID=3768916 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 04:52:43 UTC] USER=www-data EUID=0 PID=3768926 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:52:43 UTC] USER=www-data EUID=0 PID=3768935 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:52:43 UTC] USER=www-data EUID=0 PID=3768949 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 04:52:43 UTC] USER=www-data EUID=0 PID=3768982 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:52:43 UTC] USER=www-data EUID=0 PID=3769000 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:52:43 UTC] USER=www-data EUID=0 PID=3769009 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:52:43 UTC] USER=www-data EUID=0 PID=3769018 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 04:52:43 UTC] USER=www-data EUID=0 PID=3769027 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
✅ Symlinked client-key.pem
[2026-02-05 04:52:43 UTC] USER=www-data EUID=0 PID=3769045 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:52:43 UTC] USER=www-data EUID=0 PID=3769066 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:52:44 UTC] USER=www-data EUID=0 PID=3769078 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: postgres
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: postgres
Identifier: coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: coordinator
User (CN): postgres
Hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 04:52:44 UTC] USER=www-data EUID=0 PID=3769128 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 04:52:44 UTC] USER=www-data EUID=0 PID=3769144 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-02-05 04:52:44 UTC] USER=www-data EUID=0 PID=3769153 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 04:52:44 UTC] USER=www-data EUID=0 PID=3769162 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:52:45 UTC] USER=www-data EUID=0 PID=3769207 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:52:45 UTC] USER=www-data EUID=0 PID=3769226 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 04:52:45 UTC] USER=www-data EUID=0 PID=3769235 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 04:52:45 UTC] USER=www-data EUID=0 PID=3769278 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 04:52:45 UTC] USER=www-data EUID=0 PID=3769298 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 04:52:45 UTC] USER=www-data EUID=0 PID=3769311 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 04:52:45 UTC] USER=www-data EUID=0 PID=3769320 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 04:52:45 UTC] USER=www-data EUID=0 PID=3769329 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 04:52:45 UTC] USER=www-data EUID=0 PID=3769347 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:52:45 UTC] USER=www-data EUID=0 PID=3769356 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:52:45 UTC] USER=www-data EUID=0 PID=3769365 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769374 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769383 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769392 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769401 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769410 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769437 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769446 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769455 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769464 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769473 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769482 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769491 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769500 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769509 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769518 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769527 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769536 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769546 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769556 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769584 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769602 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769611 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769620 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769629 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:52:46 UTC] USER=www-data EUID=0 PID=3769638 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769649 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769668 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769678 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769698 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769719 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769750 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769760 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769769 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769778 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769787 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769796 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769806 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769816 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769834 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769843 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769852 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769862 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769881 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769890 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769899 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 04:52:47 UTC] USER=www-data EUID=0 PID=3769920 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 04:52:48 UTC] USER=www-data EUID=0 PID=3769930 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:52:48 UTC] USER=www-data EUID=0 PID=3769958 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 04:52:48 UTC] USER=www-data EUID=0 PID=3769967 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
✅ Symlinked client-key.pem
[2026-02-05 04:52:48 UTC] USER=www-data EUID=0 PID=3769985 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:52:48 UTC] USER=www-data EUID=0 PID=3769994 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:52:48 UTC] USER=www-data EUID=0 PID=3770003 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: postgres
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres
[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=752a811e-1a02-4989-b8c6-1afd6a8f211b)
[INFO] 📦 02 setup pg instance...
[0;34m[DEADLOCK-PREVENTION][0m Deadlock prevention library loaded
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Using existing db-coordinator-postgresql environment: db-web-universe-main-dev-postgresql-coordinator.fastorder.com (10.100.1.119)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.119
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m Data dir: /data/postgresql/17/web-universe-main-dev/coordinator
[0;34m[INFO][0m Port: 5432
[0;34m[INFO][0m Hostname: db-web-universe-main-dev-postgresql-coordinator
[2026-02-05 04:52:50 UTC] USER=www-data EUID=0 PID=3770176 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:52:50 UTC] USER=www-data EUID=0 PID=3770200 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:52:50 UTC] USER=www-data EUID=0 PID=3770222 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:52:50 UTC] USER=www-data EUID=0 PID=3770264 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[1;33m[WARN][0m Server certificate not found at /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[0;34m[INFO][0m Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe (FastOrder Universe)
Branch: main
Env: dev
Node: coordinator
Primary CN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Alt CN: web-universe-main-dev.fastorder.com
VM IP: 142.93.238.16
Coordinator variants:
- db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
- db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator and /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:52:51 UTC] USER=www-data EUID=0 PID=3770327 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-02-05 04:52:51 UTC] USER=www-data EUID=0 PID=3770337 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-3770275
[2026-02-05 04:52:51 UTC] USER=www-data EUID=0 PID=3770347 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-3770275/ra_root.crt
[2026-02-05 04:52:51 UTC] USER=www-data EUID=0 PID=3770356 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-3770275/ra_root.key
[2026-02-05 04:52:51 UTC] USER=www-data EUID=0 PID=3770368 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3770275/ra_root.crt
[2026-02-05 04:52:51 UTC] USER=www-data EUID=0 PID=3770383 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3770275/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[2026-02-05 04:52:54 UTC] USER=www-data EUID=0 PID=3770505 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3770275/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 04:52:54 UTC] USER=www-data EUID=0 PID=3770518 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-02-05 04:52:54 UTC] USER=www-data EUID=0 PID=3770533 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3770275/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:52:54 UTC] USER=www-data EUID=0 PID=3770543 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:52:54 UTC] USER=www-data EUID=0 PID=3770552 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 04:52:54 UTC] USER=www-data EUID=0 PID=3770573 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 04:52:54 UTC] USER=www-data EUID=0 PID=3770582 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 04:52:54 UTC] USER=www-data EUID=0 PID=3770594 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 04:52:55 UTC] USER=www-data EUID=0 PID=3770622 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔍 Verifying certificate...
Certificate details:
Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
--
X509v3 Subject Alternative Name:
DNS:db-web-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:db-web-universe-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-web-universe-main-dev-postgresql.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
X509v3 Subject Key Identifier:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Node: coordinator
Primary CN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-coordinator.service
3. Test SSL connection:
psql "host=db-web-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-02-05 04:52:55 UTC] USER=www-data EUID=0 PID=3770666 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 04:52:55 UTC] USER=www-data EUID=0 PID=3770676 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 04:52:55 UTC] USER=www-data EUID=0 PID=3770685 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[0;32m[OK][0m mTLS certificates OK (server cert + client certs verified) and keys secured
[0;34m[INFO][0m Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-02-05 04:52:55 UTC] USER=www-data EUID=0 PID=3770707 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 04:52:55 UTC] USER=www-data EUID=0 PID=3770734 ACTION=passthru ARGS=systemctl stop postgresql
[0;32m[OK][0m No conflicting Postgres left on port 5432
[0;32m[OK][0m Generated new postgres password for initdb
[2026-02-05 04:53:19 UTC] USER=www-data EUID=0 PID=3771768 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.jl6ijX
[2026-02-05 04:53:19 UTC] USER=www-data EUID=0 PID=3771791 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.jl6ijX
[2026-02-05 04:53:19 UTC] USER=www-data EUID=0 PID=3771813 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev
[2026-02-05 04:53:19 UTC] USER=www-data EUID=0 PID=3771839 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev
[2026-02-05 04:53:19 UTC] USER=www-data EUID=0 PID=3771863 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/web-universe-main-dev
[0;34m[INFO][0m Initializing cluster in /data/postgresql/17/web-universe-main-dev/coordinator (SCRAM; pwfile)
[2026-02-05 04:53:19 UTC] USER=www-data EUID=0 PID=3771885 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 04:53:20 UTC] USER=www-data EUID=0 PID=3771906 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 04:53:20 UTC] USER=www-data EUID=0 PID=3771930 ACTION=fsop ARGS=chmod 700 /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 04:53:20 UTC] USER=www-data EUID=0 PID=3771958 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-web-universe-main-dev-coordinator
[2026-02-05 04:53:20 UTC] USER=www-data EUID=0 PID=3771979 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-web-universe-main-dev-coordinator
[2026-02-05 04:53:20 UTC] USER=www-data EUID=0 PID=3772000 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-web-universe-main-dev-coordinator
[2026-02-05 04:53:20 UTC] USER=www-data EUID=0 PID=3772009 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/web-universe-main-dev/coordinator --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.jl6ijX
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /data/postgresql/17/web-universe-main-dev/coordinator ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
/usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/web-universe-main-dev/coordinator -l logfile start
[0;32m[OK][0m initdb complete
[2026-02-05 04:53:21 UTC] USER=www-data EUID=0 PID=3772106 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.jl6ijX
[0;34m[INFO][0m Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[0;32m[OK][0m postgresql.conf updated successfully
[0;34m[INFO][0m Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-02-05 04:53:21 UTC] USER=www-data EUID=0 PID=3772169 ACTION=fsop ARGS=cp /tmp/tmp.Bf5FPajgXm /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
[2026-02-05 04:53:22 UTC] USER=www-data EUID=0 PID=3772193 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
[2026-02-05 04:53:22 UTC] USER=www-data EUID=0 PID=3772220 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
[0;32m[OK][0m pg_hba.conf updated
[0;34m[INFO][0m Creating systemd unit: /etc/systemd/system/postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 04:53:22 UTC] USER=www-data EUID=0 PID=3772252 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.GLEQME /etc/systemd/system/postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 04:53:22 UTC] USER=www-data EUID=0 PID=3772273 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m systemd unit written
[2026-02-05 04:53:22 UTC] USER=www-data EUID=0 PID=3772296 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 04:53:22 UTC] USER=www-data EUID=0 PID=3772317 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 04:53:22 UTC] USER=www-data EUID=0 PID=3772338 ACTION=passthru ARGS=systemctl daemon-reload
[0;34m[INFO][0m Starting PostgreSQL instance...
[2026-02-05 04:53:23 UTC] USER=www-data EUID=0 PID=3772501 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[0;34m[INFO][0m Waiting for ACTIVE (systemd)…
[0;32m[OK][0m Service ACTIVE
[0;34m[INFO][0m Waiting for port 5432 bind…
[0;32m[OK][0m Port bound
[0;34m[INFO][0m Waiting pg_isready (socket)…
[0;32m[OK][0m Readiness via socket OK
[0;34m[INFO][0m Waiting pg_isready (TCP db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432)…
[0;32m[OK][0m Startup sequence complete
[0;34m[INFO][0m Validating core security GUCs (via local socket)…
[0;32m[OK][0m Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[0;34m[INFO][0m Provisioning application database and Debezium role (if not exists)...
[0;34m[INFO][0m Checking if database fastorder_web_universe_main_dev_db exists...
[0;34m[INFO][0m DB check result: exit_code=0, output='[2026-02-05 04:53:25 UTC] USER=www-data EUID=0 PID=3772741 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_web_universe_main_dev_db''
[0;34m[INFO][0m Creating database fastorder_web_universe_main_dev_db...
[2026-02-05 04:53:26 UTC] USER=www-data EUID=0 PID=3772764 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_web_universe_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[0;32m[OK][0m Database fastorder_web_universe_main_dev_db created
[0;34m[INFO][0m Checking if role debezium_user exists...
[0;34m[INFO][0m Role check result: exit_code=0, output='[2026-02-05 04:53:26 UTC] USER=www-data EUID=0 PID=3772806 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[0;34m[INFO][0m Creating role debezium_user...
[2026-02-05 04:53:26 UTC] USER=www-data EUID=0 PID=3772839 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'Mx1X4DX6uZqksElo4wzN5BLG';
CREATE ROLE
[0;32m[OK][0m Role debezium_user created
[2026-02-05 04:53:26 UTC] USER=www-data EUID=0 PID=3772865 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_web_universe_main_dev_db" TO debezium_user;
GRANT
[0;32m[OK][0m Application DB (fastorder_web_universe_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[0;34m[INFO][0m Applying connection and memory optimizations...
[0;34m[INFO][0m Current settings: max_connections=100, work_mem=4MB
[0;34m[INFO][0m Target settings (coordinator): max_connections=150, work_mem=8MB
[2026-02-05 04:53:27 UTC] USER=www-data EUID=0 PID=3772951 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 150;
ALTER SYSTEM
[2026-02-05 04:53:27 UTC] USER=www-data EUID=0 PID=3772975 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-02-05 04:53:27 UTC] USER=www-data EUID=0 PID=3773003 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
pg_reload_conf
----------------
t
(1 row)
[0;32m[OK][0m Settings applied to postgresql.auto.conf
[2026-02-05 04:53:28 UTC] USER=www-data EUID=0 PID=3773036 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/coordinator/standby.signal
[0;34m[INFO][0m Service recently started (4s ago) - restarting to apply max_connections...
[0;34m[INFO][0m Stopping service...
[2026-02-05 04:53:28 UTC] USER=www-data EUID=0 PID=3773058 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[0;34m[INFO][0m Waiting for port 5432 to be released...
[0;32m[OK][0m Port 5432 released
[0;34m[INFO][0m Starting service...
[2026-02-05 04:53:31 UTC] USER=www-data EUID=0 PID=3773188 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 04:53:37 UTC] USER=www-data EUID=0 PID=3773363 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m ✅ Optimization complete: max_connections=150, work_mem=8MB
[0;34m[INFO][0m Setting postgres password via centralized script... for coordinator
[0;34m[INFO][0m Temporarily disabling synchronous_commit on coordinator for password setting...
[0;32m[OK][0m Disabled synchronous_commit (was: on)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: coordinator[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-coordinator[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mInitial setup: Using password from initdb[0m
[0;32m✓ PostgreSQL password already set during initdb[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator
✅ Secret created: fastorder/db/web/universe/main/dev/postgresql/coordinator
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/coordinator[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials coordinator[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
[0;34m[INFO][0m Restoring synchronous_commit on coordinator...
[0;32m[OK][0m Restored synchronous_commit to: on
[0;32m[OK][0m Password set and persisted
[0;34m[INFO][0m Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.119
[0;34m[INFO][0m Primary hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[0;34m[INFO][0m Adding /etc/hosts entries for coordinator...
[0;34m[INFO][0m 1. db-web-universe-main-dev-postgresql.fastorder.com → 10.100.1.119 (primary/short)
[0;34m[INFO][0m 2. db-web-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.119 (compatibility)
[0;34m[INFO][0m ✅ db-web-universe-main-dev-postgresql.fastorder.com already exists with correct IP
[0;34m[INFO][0m ✅ db-web-universe-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
10.100.1.119 db-web-universe-main-dev-postgresql.fastorder.com
10.100.1.119 db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[0;32m[OK][0m PostgreSQL 'web-universe-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt \
sslcert=/home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt \
sslkey=/home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key \
host=db-web-universe-main-dev-postgresql-coordinator port=5432 dbname=postgres user=postgres"
File been compeleted perfectly: 02-setup-pg-instance
[0;34m[INFO][0m Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: PostgreSQL
[INFO] Identifier: web-universe-main-dev-postgresql-coordinator
[INFO] Identifier Parent: coordinator
[INFO] IP: 10.100.1.119
[INFO] Port: 5432
[INFO] FQDN: db-web-universe-main-dev-postgresql-coordinator
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 90feffd7-89fb-4afb-a63f-cc975d7e928c
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[0;32m[OK][0m PostgreSQL node registered to observability API
[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=752a811e-1a02-4989-b8c6-1afd6a8f211b)
[INFO] 📦 03 role...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 04:53:49 UTC] USER=www-data EUID=0 PID=3774146 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/coordinator/standby.signal
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: debezium_user
Identifier: coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: coordinator
User (CN): debezium_user
Hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 04:53:50 UTC] USER=www-data EUID=0 PID=3774319 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-debezium_user
[2026-02-05 04:53:50 UTC] USER=www-data EUID=0 PID=3774328 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-02-05 04:53:50 UTC] USER=www-data EUID=0 PID=3774338 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-debezium_user/ra_root.key
[2026-02-05 04:53:50 UTC] USER=www-data EUID=0 PID=3774348 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-02-05 04:53:50 UTC] USER=www-data EUID=0 PID=3774357 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774372 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774381 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774390 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774399 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774408 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774417 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774426 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774435 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774445 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774457 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774467 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774476 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774485 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774494 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774503 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774512 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774521 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774530 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774556 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774565 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774575 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774584 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774593 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774602 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774611 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774620 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774629 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774638 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774647 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774656 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774666 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774676 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774685 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774694 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774705 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:53:51 UTC] USER=www-data EUID=0 PID=3774715 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774724 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774733 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774772 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774781 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774790 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774799 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774809 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774819 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774828 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774837 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774846 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774856 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774866 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774876 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774886 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774895 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774904 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774916 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774925 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774941 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774953 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774962 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774973 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774982 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3774991 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3775000 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3775009 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3775018 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3775027 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3775036 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3775045 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3775054 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3775065 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3775075 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:53:52 UTC] USER=www-data EUID=0 PID=3775087 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:53:53 UTC] USER=www-data EUID=0 PID=3775099 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 04:53:53 UTC] USER=www-data EUID=0 PID=3775108 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 04:53:53 UTC] USER=www-data EUID=0 PID=3775117 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 04:53:53 UTC] USER=www-data EUID=0 PID=3775126 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:53:53 UTC] USER=www-data EUID=0 PID=3775139 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:53:53 UTC] USER=www-data EUID=0 PID=3775149 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:53:53 UTC] USER=www-data EUID=0 PID=3775159 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: debezium_user
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U debezium_user -d postgres
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: coordinator[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-coordinator[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mGenerating new secure password...[0m
[0;34mUser fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: fastorder_admin_gd[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Secret created: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials coordinator[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: coordinator/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-web-universe-main-dev-postgresql.fastorder.com
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: fastorder_admin_gd
Identifier: coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: coordinator
User (CN): fastorder_admin_gd
Hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 04:54:02 UTC] USER=www-data EUID=0 PID=3776022 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-fastorder_admin_gd
[2026-02-05 04:54:02 UTC] USER=www-data EUID=0 PID=3776033 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-02-05 04:54:02 UTC] USER=www-data EUID=0 PID=3776042 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
[2026-02-05 04:54:02 UTC] USER=www-data EUID=0 PID=3776053 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:54:03 UTC] USER=www-data EUID=0 PID=3776102 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:54:03 UTC] USER=www-data EUID=0 PID=3776113 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:54:03 UTC] USER=www-data EUID=0 PID=3776122 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 04:54:03 UTC] USER=www-data EUID=0 PID=3776131 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 04:54:03 UTC] USER=www-data EUID=0 PID=3776141 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:54:03 UTC] USER=www-data EUID=0 PID=3776152 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:54:03 UTC] USER=www-data EUID=0 PID=3776161 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 04:54:03 UTC] USER=www-data EUID=0 PID=3776170 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 04:54:03 UTC] USER=www-data EUID=0 PID=3776179 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 04:54:03 UTC] USER=www-data EUID=0 PID=3776188 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 04:54:03 UTC] USER=www-data EUID=0 PID=3776197 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776206 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776224 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776243 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776252 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776261 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776270 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776279 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776288 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776297 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776323 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776332 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776341 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776350 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776360 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776378 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776387 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776397 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776406 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776415 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776424 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776434 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:54:04 UTC] USER=www-data EUID=0 PID=3776444 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:54:05 UTC] USER=www-data EUID=0 PID=3776453 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:05 UTC] USER=www-data EUID=0 PID=3776462 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:05 UTC] USER=www-data EUID=0 PID=3776488 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:54:05 UTC] USER=www-data EUID=0 PID=3776503 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 04:54:05 UTC] USER=www-data EUID=0 PID=3776515 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 04:54:05 UTC] USER=www-data EUID=0 PID=3776524 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:54:05 UTC] USER=www-data EUID=0 PID=3776533 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:54:05 UTC] USER=www-data EUID=0 PID=3776542 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 04:54:05 UTC] USER=www-data EUID=0 PID=3776555 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 04:54:05 UTC] USER=www-data EUID=0 PID=3776564 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 04:54:05 UTC] USER=www-data EUID=0 PID=3776575 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:54:05 UTC] USER=www-data EUID=0 PID=3776591 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:54:05 UTC] USER=www-data EUID=0 PID=3776603 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:05 UTC] USER=www-data EUID=0 PID=3776612 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:05 UTC] USER=www-data EUID=0 PID=3776621 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:54:05 UTC] USER=www-data EUID=0 PID=3776639 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 04:54:06 UTC] USER=www-data EUID=0 PID=3776653 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 04:54:06 UTC] USER=www-data EUID=0 PID=3776663 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:54:06 UTC] USER=www-data EUID=0 PID=3776675 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:54:06 UTC] USER=www-data EUID=0 PID=3776684 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 04:54:06 UTC] USER=www-data EUID=0 PID=3776693 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 04:54:06 UTC] USER=www-data EUID=0 PID=3776702 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 04:54:06 UTC] USER=www-data EUID=0 PID=3776713 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:54:06 UTC] USER=www-data EUID=0 PID=3776724 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:54:06 UTC] USER=www-data EUID=0 PID=3776733 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:06 UTC] USER=www-data EUID=0 PID=3776752 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:54:06 UTC] USER=www-data EUID=0 PID=3776762 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 04:54:06 UTC] USER=www-data EUID=0 PID=3776771 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 04:54:06 UTC] USER=www-data EUID=0 PID=3776782 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 04:54:06 UTC] USER=www-data EUID=0 PID=3776798 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 04:54:06 UTC] USER=www-data EUID=0 PID=3776807 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 04:54:06 UTC] USER=www-data EUID=0 PID=3776816 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 04:54:07 UTC] USER=www-data EUID=0 PID=3776826 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 04:54:07 UTC] USER=www-data EUID=0 PID=3776836 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 04:54:07 UTC] USER=www-data EUID=0 PID=3776846 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:54:07 UTC] USER=www-data EUID=0 PID=3776858 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:54:07 UTC] USER=www-data EUID=0 PID=3776867 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:54:07 UTC] USER=www-data EUID=0 PID=3776876 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 04:54:07 UTC] USER=www-data EUID=0 PID=3776885 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 04:54:07 UTC] USER=www-data EUID=0 PID=3776894 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 04:54:07 UTC] USER=www-data EUID=0 PID=3776903 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:54:07 UTC] USER=www-data EUID=0 PID=3776912 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:54:07 UTC] USER=www-data EUID=0 PID=3776921 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:54:07 UTC] USER=www-data EUID=0 PID=3776930 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: fastorder_admin_gd
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U fastorder_admin_gd -d postgres
🧱 Connecting via Unix socket to create role and database...
Socket: /var/run/postgresql-web-universe-main-dev-coordinator:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️ Database fastorder_web_universe_main_dev_db already exists, skipping creation
[2026-02-05 04:54:07 UTC] USER=www-data EUID=0 PID=3777004 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️ Using pg_hba.conf: /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-02-05 04:54:07 UTC] USER=www-data EUID=0 PID=3777041 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[0;34m=== Pre-flight Checks ===[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;32m✓[0m AWS Secrets Manager accessible
[0;34m=== Retrieving Credentials from AWS ===[0m
ℹ️ Retrieving PostgreSQL credentials for: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️ Fetching secret: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials loaded for coordinator/fastorder_admin_gd: fastorder_admin_gd@db-web-universe-main-dev-postgresql.fastorder.com:5432/fastorder_web_universe_main_dev_db
[0;32m✓[0m Credentials retrieved: fastorder_admin_gd@db-web-universe-main-dev-postgresql.fastorder.com:5432/fastorder_web_universe_main_dev_db
[0;34m╔════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Test Suite (AWS Secrets MGR) ║[0m
[0;34m╚════════════════════════════════════════════╝[0m
[0;34m=== PostgreSQL Authentication Test ===[0m
[0;31m✗[0m PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-web-universe-main-dev-postgresql.fastorder.com" (10.100.1.119), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
── fast setup ─────────────────────────────────────────────
NAME : web-universe-main-dev
IDENTIFIER : coordinator
PG HOST : db-web-universe-main-dev-postgresql.fastorder.com:5432
ROLE : debezium_user
DB : fastorder_web_universe_main_dev_db
SCHEMA : web
AUTH MODE : scram (scram=password over TLS | cert=mTLS)
SUBNET ALLOW: 10.201.0.0/16
CONNECT /32 : 142.93.238.16
SSL DIR : /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
DNS → 10.100.1.119
CA : /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: coordinator[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-coordinator[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mGenerating new secure password...[0m
[0;34mUser debezium_user does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: debezium_user[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
✅ Secret created: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials coordinator[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: coordinator/debezium_user
✓ Retrieved password from secrets vault
password : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️ Role debezium_user exists, updating
[2026-02-05 04:54:22 UTC] USER=www-data EUID=0 PID=3777876 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️ Database fastorder_web_universe_main_dev_db already exists
[2026-02-05 04:54:23 UTC] USER=www-data EUID=0 PID=3777911 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d fastorder_web_universe_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️ Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=752a811e-1a02-4989-b8c6-1afd6a8f211b)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
ℹ️ Service-specific setup (web) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)
🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=web
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
🔍 DEBUG_CHECKPOINT_03: No specific folder for web, using default
[DEBUG] Tracking substep start: steps/01-install/steps/default (RUN_UUID=752a811e-1a02-4989-b8c6-1afd6a8f211b)
[INFO] 🔸 Service: web (using default contracts schema)
🔍 DEBUG_CHECKPOINT_04: Executing default: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/run.sh
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 🟢 Starting default contracts schema provisioning for SERVICE=web
[INFO] Environment: web-universe-main-dev
[INFO] Schema: web (contracts tables)
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16
🔍 DEBUG: Looking for contracts steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/default/contracts/steps
[INFO] 📁 Running contracts schema setup for: web
[INFO] 📁 Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/default/contracts/steps
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Initializing web schema (contracts tables)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Schema: web
Identifier: coordinator
Database: fastorder_web_universe_main_dev_db
Host: db-web-universe-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️ Checking database: fastorder_web_universe_main_dev_db
ℹ️ Database fastorder_web_universe_main_dev_db already exists
✅ Connected to database: fastorder_web_universe_main_dev_db
ℹ️ Checking synchronous replication configuration...
synchronous_standby_names: ''
Connected standbys: 0
ℹ️ Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
CREATE EXTENSION
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating web schema...
NOTICE: schema "web" already exists, skipping
CREATE SCHEMA
✅ Schema created
🔧 Creating contracts tables in web schema...
Creating "web".contract_key...
CREATE TABLE
Creating "web".contract_type...
CREATE TABLE
Creating "web".contracts...
CREATE TABLE
Adding columns to "web".contracts (safe migration)...
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
UPDATE 0
UPDATE 0
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
Creating "web".contract_vars...
CREATE TABLE
Creating "web".contract_datetime...
CREATE TABLE
Creating "web".contract_decimal...
CREATE TABLE
Creating "web".contract_float...
CREATE TABLE
Creating "web".contract_int...
CREATE TABLE
Creating "web".contract_json...
CREATE TABLE
Creating "web".contract_terms...
CREATE TABLE
Creating "web".contract_term_contracts...
CREATE TABLE
Creating "web".contract_term_datetime...
CREATE TABLE
Creating "web".contract_term_decimal...
CREATE TABLE
Creating "web".contract_term_float...
CREATE TABLE
Creating "web".contract_term_int...
CREATE TABLE
Creating "web".contract_term_items...
CREATE TABLE
Creating "web".contract_term_json...
CREATE TABLE
Creating "web".contract_term_vars...
CREATE TABLE
Creating "web".web_id_uuid_mapping...
CREATE TABLE
✅ All 19 tables created
🔧 Creating indexes...
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
✅ All indexes created
🔧 Creating foreign keys...
DO
DO
✅ Foreign keys created
🔧 Configuring Citus distribution...
Creating reference table: contract_key
create_reference_table
------------------------
(1 row)
Creating reference table: contract_type
create_reference_table
------------------------
(1 row)
Creating distributed table: contracts
Creating distributed table: contract_vars
Creating distributed table: contract_datetime
Creating distributed table: contract_decimal
Creating distributed table: contract_float
Creating distributed table: contract_int
Creating distributed table: contract_json
Creating distributed table: contract_terms
Creating distributed table: contract_term_contracts
Creating distributed table: contract_term_datetime
Creating distributed table: contract_term_decimal
Creating distributed table: contract_term_float
Creating distributed table: contract_term_int
Creating distributed table: contract_term_items
Creating distributed table: contract_term_json
create_distributed_table
--------------------------
(1 row)
Creating distributed table: contract_term_vars
create_distributed_table
--------------------------
(1 row)
✅ Citus distribution configured
🎉 Schema initialization complete for web in fastorder_web_universe_main_dev_db
ℹ️ Skipping LISTEN/NOTIFY trigger on coordinator
CDC via Debezium is the primary change tracking mechanism
==========================================
✅ web schema initialization complete!
Tables: 19
Indexes: 54
==========================================
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Default contracts schema setup complete for: web
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✓[0m ✅ Coordinator setup completed
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up 1 worker(s) (Citus data nodes)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up worker: worker-01
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-02-05 04:54:42 UTC] USER=unknown EUID=33 PID=3778972 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 04:54:42 UTC] USER=unknown EUID=33 PID=3778979 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 04:54:42 UTC] USER=unknown EUID=33 PID=3778991 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-02-05 04:54:42 UTC] USER=unknown EUID=33 PID=3778998 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 04:54:42 UTC] USER=unknown EUID=33 PID=3779005 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-02-05 04:54:42 UTC] USER=unknown EUID=33 PID=3779012 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for web in universe-dev...
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=752a811e-1a02-4989-b8c6-1afd6a8f211b JOB_UUID=1afa4e23-6aff-4492-82ab-adcbd43dadaa
[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=752a811e-1a02-4989-b8c6-1afd6a8f211b)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[WARN] Could not find PostgreSQL IP for worker-01 in topology.json, allocating new VM IP...
[INFO] Allocated new VM IP: 10.100.1.243 for db-worker-01-postgresql
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.243
[INFO] Primary hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[INFO] Adding /etc/hosts entry for worker-01...
[INFO] db-web-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.243
[INFO] ➕ Adding db-web-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.243
[0;32m✅[0m ✅ Added: db-web-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.243
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
10.100.1.243 db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=752a811e-1a02-4989-b8c6-1afd6a8f211b)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe (FastOrder Universe)
Branch: main
Env: dev
Node: worker-01
Primary CN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Alt CN: web-universe-main-dev.fastorder.com
VM IP: 142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 04:54:46 UTC] USER=www-data EUID=0 PID=3779718 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01 and /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:54:46 UTC] USER=www-data EUID=0 PID=3779733 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-02-05 04:54:46 UTC] USER=www-data EUID=0 PID=3779750 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-3779685
[2026-02-05 04:54:46 UTC] USER=www-data EUID=0 PID=3779759 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-3779685/ra_root.crt
[2026-02-05 04:54:46 UTC] USER=www-data EUID=0 PID=3779768 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-3779685/ra_root.key
[2026-02-05 04:54:46 UTC] USER=www-data EUID=0 PID=3779778 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3779685/ra_root.crt
[2026-02-05 04:54:46 UTC] USER=www-data EUID=0 PID=3779787 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3779685/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[2026-02-05 04:54:47 UTC] USER=www-data EUID=0 PID=3779841 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3779685/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 04:54:47 UTC] USER=www-data EUID=0 PID=3779850 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3779685/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 04:54:47 UTC] USER=www-data EUID=0 PID=3779859 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-02-05 04:54:47 UTC] USER=www-data EUID=0 PID=3779868 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3779685/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:54:47 UTC] USER=www-data EUID=0 PID=3779878 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:54:47 UTC] USER=www-data EUID=0 PID=3779887 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:54:47 UTC] USER=www-data EUID=0 PID=3779898 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 04:54:47 UTC] USER=www-data EUID=0 PID=3779909 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 04:54:47 UTC] USER=www-data EUID=0 PID=3779918 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 04:54:47 UTC] USER=www-data EUID=0 PID=3779927 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 04:54:47 UTC] USER=www-data EUID=0 PID=3779936 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 04:54:48 UTC] USER=www-data EUID=0 PID=3779945 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:54:48 UTC] USER=www-data EUID=0 PID=3779954 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔍 Verifying certificate...
Certificate details:
Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
--
X509v3 Subject Alternative Name:
DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
X509v3 Subject Key Identifier:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Node: worker-01
Primary CN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-worker-01.service
3. Test SSL connection:
psql "host=db-web-universe-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: postgres
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: worker-01
User (CN): postgres
Hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 04:54:48 UTC] USER=www-data EUID=0 PID=3780009 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-02-05 04:54:48 UTC] USER=www-data EUID=0 PID=3780018 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 04:54:48 UTC] USER=www-data EUID=0 PID=3780027 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-02-05 04:54:48 UTC] USER=www-data EUID=0 PID=3780055 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 04:54:48 UTC] USER=www-data EUID=0 PID=3780064 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:54:48 UTC] USER=www-data EUID=0 PID=3780080 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:54:48 UTC] USER=www-data EUID=0 PID=3780089 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:54:48 UTC] USER=www-data EUID=0 PID=3780098 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 04:54:48 UTC] USER=www-data EUID=0 PID=3780107 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 04:54:48 UTC] USER=www-data EUID=0 PID=3780116 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:54:48 UTC] USER=www-data EUID=0 PID=3780125 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780134 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780143 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780152 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780161 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780170 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780179 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780188 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780198 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780218 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780237 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780250 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780261 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780299 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780311 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780321 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780330 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780372 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780381 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780390 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780399 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780408 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780418 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 04:54:49 UTC] USER=www-data EUID=0 PID=3780429 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780453 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780462 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780474 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780483 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780496 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780507 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780516 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780525 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780534 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780543 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780553 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780563 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780573 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780582 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780591 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780600 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780609 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780618 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780627 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780636 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780645 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780656 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780665 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780674 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780685 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780695 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780707 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780717 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780726 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780737 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780746 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 04:54:50 UTC] USER=www-data EUID=0 PID=3780755 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 04:54:51 UTC] USER=www-data EUID=0 PID=3780764 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:54:51 UTC] USER=www-data EUID=0 PID=3780773 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:54:51 UTC] USER=www-data EUID=0 PID=3780782 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 04:54:51 UTC] USER=www-data EUID=0 PID=3780791 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 04:54:51 UTC] USER=www-data EUID=0 PID=3780800 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 04:54:51 UTC] USER=www-data EUID=0 PID=3780810 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:54:51 UTC] USER=www-data EUID=0 PID=3780820 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:54:51 UTC] USER=www-data EUID=0 PID=3780830 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:54:51 UTC] USER=www-data EUID=0 PID=3780839 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
✅ Symlinked client-cert.pem
✅ Symlinked client-key.pem
[2026-02-05 04:54:51 UTC] USER=www-data EUID=0 PID=3780868 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:54:51 UTC] USER=www-data EUID=0 PID=3780887 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:54:51 UTC] USER=www-data EUID=0 PID=3780896 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:54:51 UTC] USER=www-data EUID=0 PID=3780905 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: postgres
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: postgres
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: worker-01
User (CN): postgres
Hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 04:54:51 UTC] USER=www-data EUID=0 PID=3780969 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-02-05 04:54:51 UTC] USER=www-data EUID=0 PID=3780987 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781000 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781015 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781033 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781065 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781074 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781085 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781097 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781112 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781122 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781131 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781140 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781150 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781159 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781168 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781177 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781186 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781195 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781204 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781215 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781227 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781236 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781245 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 04:54:52 UTC] USER=www-data EUID=0 PID=3781254 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781263 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781289 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781298 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781309 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781319 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781334 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781346 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781355 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781364 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781373 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781382 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781391 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781400 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781410 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781420 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781429 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781455 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781464 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781473 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781484 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781493 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781502 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781521 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781530 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781539 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 04:54:53 UTC] USER=www-data EUID=0 PID=3781554 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781576 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781587 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781616 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781625 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781634 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781644 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781655 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781665 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781674 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781684 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781694 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781703 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781712 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781721 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781730 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781752 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781761 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781770 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781779 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781797 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781806 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781816 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781827 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781845 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781854 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781863 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781872 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:54:54 UTC] USER=www-data EUID=0 PID=3781881 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:54:55 UTC] USER=www-data EUID=0 PID=3781890 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:54:55 UTC] USER=www-data EUID=0 PID=3781901 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: postgres
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres
[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=752a811e-1a02-4989-b8c6-1afd6a8f211b)
[INFO] 📦 02 setup pg instance...
[0;34m[DEADLOCK-PREVENTION][0m Deadlock prevention library loaded
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Using existing db-worker-01-postgresql environment: db-web-universe-main-dev-postgresql-worker-01.fastorder.com (10.100.1.243)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.243
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m Data dir: /data/postgresql/17/web-universe-main-dev/worker-01
[0;34m[INFO][0m Port: 5432
[0;34m[INFO][0m Hostname: db-web-universe-main-dev-postgresql-worker-01
[2026-02-05 04:54:56 UTC] USER=www-data EUID=0 PID=3782106 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:54:56 UTC] USER=www-data EUID=0 PID=3782127 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:54:56 UTC] USER=www-data EUID=0 PID=3782153 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[1;33m[WARN][0m Server certificate not found at /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[0;34m[INFO][0m Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe (FastOrder Universe)
Branch: main
Env: dev
Node: worker-01
Primary CN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Alt CN: web-universe-main-dev.fastorder.com
VM IP: 142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 04:54:57 UTC] USER=www-data EUID=0 PID=3782228 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01 and /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:54:57 UTC] USER=www-data EUID=0 PID=3782237 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-02-05 04:54:57 UTC] USER=www-data EUID=0 PID=3782247 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-3782183
[2026-02-05 04:54:57 UTC] USER=www-data EUID=0 PID=3782256 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-3782183/ra_root.crt
[2026-02-05 04:54:57 UTC] USER=www-data EUID=0 PID=3782265 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-3782183/ra_root.key
[2026-02-05 04:54:58 UTC] USER=www-data EUID=0 PID=3782274 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3782183/ra_root.crt
[2026-02-05 04:54:58 UTC] USER=www-data EUID=0 PID=3782283 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3782183/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[2026-02-05 04:55:00 UTC] USER=www-data EUID=0 PID=3782398 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3782183/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 04:55:00 UTC] USER=www-data EUID=0 PID=3782410 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3782183/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 04:55:00 UTC] USER=www-data EUID=0 PID=3782434 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-02-05 04:55:00 UTC] USER=www-data EUID=0 PID=3782444 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3782183/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:55:00 UTC] USER=www-data EUID=0 PID=3782456 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:55:00 UTC] USER=www-data EUID=0 PID=3782465 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:55:00 UTC] USER=www-data EUID=0 PID=3782474 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 04:55:00 UTC] USER=www-data EUID=0 PID=3782487 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 04:55:01 UTC] USER=www-data EUID=0 PID=3782497 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 04:55:01 UTC] USER=www-data EUID=0 PID=3782537 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:55:01 UTC] USER=www-data EUID=0 PID=3782571 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔍 Verifying certificate...
Certificate details:
Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
--
X509v3 Subject Alternative Name:
DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
X509v3 Subject Key Identifier:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Node: worker-01
Primary CN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-worker-01.service
3. Test SSL connection:
psql "host=db-web-universe-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-02-05 04:55:01 UTC] USER=www-data EUID=0 PID=3782607 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 04:55:02 UTC] USER=www-data EUID=0 PID=3782623 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 04:55:02 UTC] USER=www-data EUID=0 PID=3782674 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[0;32m[OK][0m mTLS certificates OK (server cert + client certs verified) and keys secured
[0;34m[INFO][0m Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-02-05 04:55:02 UTC] USER=www-data EUID=0 PID=3782776 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 04:55:02 UTC] USER=www-data EUID=0 PID=3782805 ACTION=passthru ARGS=systemctl stop postgresql
[0;32m[OK][0m No conflicting Postgres left on port 5432
[0;32m[OK][0m Generated new postgres password for initdb
[2026-02-05 04:55:27 UTC] USER=www-data EUID=0 PID=3783609 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.ApHGjZ
[2026-02-05 04:55:27 UTC] USER=www-data EUID=0 PID=3783652 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.ApHGjZ
[2026-02-05 04:55:27 UTC] USER=www-data EUID=0 PID=3783691 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev
[2026-02-05 04:55:27 UTC] USER=www-data EUID=0 PID=3783720 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev
[2026-02-05 04:55:27 UTC] USER=www-data EUID=0 PID=3783752 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/web-universe-main-dev
[0;34m[INFO][0m Initializing cluster in /data/postgresql/17/web-universe-main-dev/worker-01 (SCRAM; pwfile)
[2026-02-05 04:55:27 UTC] USER=www-data EUID=0 PID=3783774 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 04:55:27 UTC] USER=www-data EUID=0 PID=3783796 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 04:55:27 UTC] USER=www-data EUID=0 PID=3783817 ACTION=fsop ARGS=chmod 700 /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 04:55:28 UTC] USER=www-data EUID=0 PID=3783839 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-web-universe-main-dev-worker-01
[2026-02-05 04:55:28 UTC] USER=www-data EUID=0 PID=3783863 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-web-universe-main-dev-worker-01
[2026-02-05 04:55:28 UTC] USER=www-data EUID=0 PID=3783892 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-web-universe-main-dev-worker-01
[2026-02-05 04:55:28 UTC] USER=www-data EUID=0 PID=3783901 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/web-universe-main-dev/worker-01 --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.ApHGjZ
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /data/postgresql/17/web-universe-main-dev/worker-01 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
/usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/web-universe-main-dev/worker-01 -l logfile start
[0;32m[OK][0m initdb complete
[2026-02-05 04:55:29 UTC] USER=www-data EUID=0 PID=3783953 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.ApHGjZ
[0;34m[INFO][0m Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[0;32m[OK][0m postgresql.conf updated successfully
[0;34m[INFO][0m Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-02-05 04:55:29 UTC] USER=www-data EUID=0 PID=3784002 ACTION=fsop ARGS=cp /tmp/tmp.qhprGK7kqB /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
[2026-02-05 04:55:29 UTC] USER=www-data EUID=0 PID=3784041 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
[2026-02-05 04:55:29 UTC] USER=www-data EUID=0 PID=3784062 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
[0;32m[OK][0m pg_hba.conf updated
[0;34m[INFO][0m Creating systemd unit: /etc/systemd/system/postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 04:55:29 UTC] USER=www-data EUID=0 PID=3784087 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.X4Oemz /etc/systemd/system/postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 04:55:29 UTC] USER=www-data EUID=0 PID=3784109 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@web-universe-main-dev-worker-01.service
[0;32m[OK][0m systemd unit written
[2026-02-05 04:55:29 UTC] USER=www-data EUID=0 PID=3784130 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 04:55:29 UTC] USER=www-data EUID=0 PID=3784151 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 04:55:30 UTC] USER=www-data EUID=0 PID=3784175 ACTION=passthru ARGS=systemctl daemon-reload
[0;34m[INFO][0m Starting PostgreSQL instance...
[2026-02-05 04:55:31 UTC] USER=www-data EUID=0 PID=3784349 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-worker-01.service
[0;34m[INFO][0m Waiting for ACTIVE (systemd)…
[2026-02-05 04:55:31 UTC] USER=www-data EUID=0 PID=3784390 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-worker-01.service
[0;32m[OK][0m Service ACTIVE
[0;34m[INFO][0m Waiting for port 5432 bind…
[0;32m[OK][0m Port bound
[0;34m[INFO][0m Waiting pg_isready (socket)…
[0;32m[OK][0m Readiness via socket OK
[0;34m[INFO][0m Waiting pg_isready (TCP db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432)…
[0;32m[OK][0m Startup sequence complete
[0;34m[INFO][0m Validating core security GUCs (via local socket)…
[0;32m[OK][0m Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[0;34m[INFO][0m Provisioning application database and Debezium role (if not exists)...
[0;34m[INFO][0m Checking if database fastorder_web_universe_main_dev_db exists...
[0;34m[INFO][0m DB check result: exit_code=0, output='[2026-02-05 04:55:32 UTC] USER=www-data EUID=0 PID=3784549 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_web_universe_main_dev_db''
[0;34m[INFO][0m Creating database fastorder_web_universe_main_dev_db...
[2026-02-05 04:55:32 UTC] USER=www-data EUID=0 PID=3784580 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_web_universe_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[0;32m[OK][0m Database fastorder_web_universe_main_dev_db created
[0;34m[INFO][0m Checking if role debezium_user exists...
[0;34m[INFO][0m Role check result: exit_code=0, output='[2026-02-05 04:55:32 UTC] USER=www-data EUID=0 PID=3784619 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[0;34m[INFO][0m Creating role debezium_user...
[2026-02-05 04:55:33 UTC] USER=www-data EUID=0 PID=3784652 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'OjPijORtHOsmGEogpEIFSZWx';
CREATE ROLE
[0;32m[OK][0m Role debezium_user created
[2026-02-05 04:55:33 UTC] USER=www-data EUID=0 PID=3784676 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_web_universe_main_dev_db" TO debezium_user;
GRANT
[0;32m[OK][0m Application DB (fastorder_web_universe_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[0;34m[INFO][0m Applying connection and memory optimizations...
[0;34m[INFO][0m Current settings: max_connections=100, work_mem=4MB
[0;34m[INFO][0m Target settings (worker): max_connections=100, work_mem=8MB
[2026-02-05 04:55:33 UTC] USER=www-data EUID=0 PID=3784755 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 100;
ALTER SYSTEM
[2026-02-05 04:55:33 UTC] USER=www-data EUID=0 PID=3784779 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-02-05 04:55:33 UTC] USER=www-data EUID=0 PID=3784802 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
pg_reload_conf
----------------
t
(1 row)
[0;32m[OK][0m Settings applied to postgresql.auto.conf
[2026-02-05 04:55:33 UTC] USER=www-data EUID=0 PID=3784817 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/worker-01/standby.signal
[0;34m[INFO][0m Service recently started (2s ago) - restarting to apply max_connections...
[0;34m[INFO][0m Stopping service...
[2026-02-05 04:55:33 UTC] USER=www-data EUID=0 PID=3784840 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-worker-01.service
[0;34m[INFO][0m Waiting for port 5432 to be released...
[0;32m[OK][0m Port 5432 released
[0;34m[INFO][0m Starting service...
[2026-02-05 04:55:37 UTC] USER=www-data EUID=0 PID=3784988 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 04:55:43 UTC] USER=www-data EUID=0 PID=3785271 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-worker-01.service
[0;32m[OK][0m ✅ Optimization complete: max_connections=100, work_mem=8MB
[0;32m[OK][0m Synchronous replication already configured (synchronous_commit: on)
[0;34m[INFO][0m Setting postgres password via centralized script... for worker-01
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: worker-01[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-worker-01[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mInitial setup: Using password from initdb[0m
[0;32m✓ PostgreSQL password already set during initdb[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01
✅ Secret created: fastorder/db/web/universe/main/dev/postgresql/worker-01
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/worker-01[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials worker-01[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
[0;32m[OK][0m Password set and persisted
[0;34m[INFO][0m Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.243
[0;34m[INFO][0m Primary hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[0;34m[INFO][0m Adding /etc/hosts entry for worker-01...
[0;34m[INFO][0m db-web-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.243
[0;34m[INFO][0m ✅ db-web-universe-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
10.100.1.243 db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[0;32m[OK][0m PostgreSQL 'web-universe-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt \
sslcert=/home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt \
sslkey=/home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key \
host=db-web-universe-main-dev-postgresql-worker-01 port=5432 dbname=postgres user=postgres"
File been compeleted perfectly: 02-setup-pg-instance
[0;34m[INFO][0m Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: PostgreSQL
[INFO] Identifier: web-universe-main-dev-postgresql-worker-01
[INFO] Identifier Parent: worker-01
[INFO] IP: 10.100.1.243
[INFO] Port: 5432
[INFO] FQDN: db-web-universe-main-dev-postgresql-worker-01
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 84e4185a-2ef1-49c1-8d2a-841d077f036b
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[0;32m[OK][0m PostgreSQL node registered to observability API
[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=752a811e-1a02-4989-b8c6-1afd6a8f211b)
[INFO] 📦 03 role...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 04:55:54 UTC] USER=www-data EUID=0 PID=3785882 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/worker-01/standby.signal
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: debezium_user
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: worker-01
User (CN): debezium_user
Hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 04:55:55 UTC] USER=www-data EUID=0 PID=3786066 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-debezium_user
[2026-02-05 04:55:55 UTC] USER=www-data EUID=0 PID=3786097 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-02-05 04:55:55 UTC] USER=www-data EUID=0 PID=3786115 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-debezium_user/ra_root.key
[2026-02-05 04:55:56 UTC] USER=www-data EUID=0 PID=3786127 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-02-05 04:55:56 UTC] USER=www-data EUID=0 PID=3786137 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:55:56 UTC] USER=www-data EUID=0 PID=3786167 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:55:56 UTC] USER=www-data EUID=0 PID=3786176 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 04:55:56 UTC] USER=www-data EUID=0 PID=3786189 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 04:55:56 UTC] USER=www-data EUID=0 PID=3786198 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:55:56 UTC] USER=www-data EUID=0 PID=3786216 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 04:55:56 UTC] USER=www-data EUID=0 PID=3786225 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 04:55:56 UTC] USER=www-data EUID=0 PID=3786234 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 04:55:56 UTC] USER=www-data EUID=0 PID=3786247 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 04:55:56 UTC] USER=www-data EUID=0 PID=3786258 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:55:56 UTC] USER=www-data EUID=0 PID=3786267 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786287 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786296 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786305 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786314 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786323 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786359 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786374 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786406 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786430 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786440 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786449 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786459 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786468 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786477 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786486 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786495 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786506 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786518 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786531 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786542 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:55:57 UTC] USER=www-data EUID=0 PID=3786551 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:55:58 UTC] USER=www-data EUID=0 PID=3786597 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:55:58 UTC] USER=www-data EUID=0 PID=3786606 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:55:58 UTC] USER=www-data EUID=0 PID=3786616 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 04:55:58 UTC] USER=www-data EUID=0 PID=3786626 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 04:55:58 UTC] USER=www-data EUID=0 PID=3786635 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 04:55:58 UTC] USER=www-data EUID=0 PID=3786658 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:55:58 UTC] USER=www-data EUID=0 PID=3786674 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:55:58 UTC] USER=www-data EUID=0 PID=3786696 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:55:58 UTC] USER=www-data EUID=0 PID=3786708 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:55:58 UTC] USER=www-data EUID=0 PID=3786717 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:55:58 UTC] USER=www-data EUID=0 PID=3786726 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 04:55:58 UTC] USER=www-data EUID=0 PID=3786735 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 04:55:58 UTC] USER=www-data EUID=0 PID=3786744 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786771 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786780 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786790 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786800 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786809 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786818 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786828 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786839 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786848 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786857 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786877 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786888 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786899 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786908 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786918 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786928 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786939 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786950 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 04:55:59 UTC] USER=www-data EUID=0 PID=3786959 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
✅ Symlinked client-key.pem
[2026-02-05 04:56:00 UTC] USER=www-data EUID=0 PID=3786977 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:56:00 UTC] USER=www-data EUID=0 PID=3786995 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:56:00 UTC] USER=www-data EUID=0 PID=3787004 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: debezium_user
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U debezium_user -d postgres
🔐 Generating replicator client certificate for worker-01...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: replicator
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: worker-01
User (CN): replicator
Hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 04:56:00 UTC] USER=www-data EUID=0 PID=3787077 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-02-05 04:56:00 UTC] USER=www-data EUID=0 PID=3787086 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-02-05 04:56:00 UTC] USER=www-data EUID=0 PID=3787097 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-02-05 04:56:00 UTC] USER=www-data EUID=0 PID=3787106 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-02-05 04:56:01 UTC] USER=www-data EUID=0 PID=3787115 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:56:01 UTC] USER=www-data EUID=0 PID=3787131 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:56:01 UTC] USER=www-data EUID=0 PID=3787140 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:56:01 UTC] USER=www-data EUID=0 PID=3787158 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 04:56:01 UTC] USER=www-data EUID=0 PID=3787167 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 04:56:01 UTC] USER=www-data EUID=0 PID=3787176 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:56:01 UTC] USER=www-data EUID=0 PID=3787185 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:56:01 UTC] USER=www-data EUID=0 PID=3787194 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 04:56:01 UTC] USER=www-data EUID=0 PID=3787203 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 04:56:01 UTC] USER=www-data EUID=0 PID=3787212 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 04:56:01 UTC] USER=www-data EUID=0 PID=3787232 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 04:56:02 UTC] USER=www-data EUID=0 PID=3787259 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 04:56:02 UTC] USER=www-data EUID=0 PID=3787270 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 04:56:02 UTC] USER=www-data EUID=0 PID=3787282 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 04:56:02 UTC] USER=www-data EUID=0 PID=3787320 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:56:02 UTC] USER=www-data EUID=0 PID=3787362 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:56:02 UTC] USER=www-data EUID=0 PID=3787430 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 04:56:02 UTC] USER=www-data EUID=0 PID=3787456 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 04:56:02 UTC] USER=www-data EUID=0 PID=3787469 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 04:56:02 UTC] USER=www-data EUID=0 PID=3787480 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:56:02 UTC] USER=www-data EUID=0 PID=3787492 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:02 UTC] USER=www-data EUID=0 PID=3787518 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787536 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787550 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787559 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787568 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787578 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787592 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787612 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787621 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787631 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787645 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787661 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787678 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787696 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787708 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787726 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787739 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787748 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787757 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787766 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 04:56:03 UTC] USER=www-data EUID=0 PID=3787775 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787784 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787795 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787806 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787815 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787824 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787833 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787843 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787855 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787865 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787874 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787883 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787892 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787901 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787910 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787920 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787930 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787939 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787949 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787958 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787967 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787976 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 04:56:04 UTC] USER=www-data EUID=0 PID=3787985 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 04:56:05 UTC] USER=www-data EUID=0 PID=3787994 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:56:05 UTC] USER=www-data EUID=0 PID=3788003 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:56:05 UTC] USER=www-data EUID=0 PID=3788012 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 04:56:05 UTC] USER=www-data EUID=0 PID=3788021 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 04:56:05 UTC] USER=www-data EUID=0 PID=3788030 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 04:56:05 UTC] USER=www-data EUID=0 PID=3788040 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:56:05 UTC] USER=www-data EUID=0 PID=3788050 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:56:05 UTC] USER=www-data EUID=0 PID=3788059 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
✅ Symlinked ca.pem
[2026-02-05 04:56:05 UTC] USER=www-data EUID=0 PID=3788078 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 04:56:05 UTC] USER=www-data EUID=0 PID=3788102 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 04:56:05 UTC] USER=www-data EUID=0 PID=3788111 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:56:05 UTC] USER=www-data EUID=0 PID=3788120 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:56:05 UTC] USER=www-data EUID=0 PID=3788129 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:56:05 UTC] USER=www-data EUID=0 PID=3788139 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: replicator
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres
✅ Replicator certificate generated for worker-01
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: worker-01[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-worker-01[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mGenerating new secure password...[0m
[0;34mUser fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: fastorder_admin_gd[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Secret created: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials worker-01[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: worker-01/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: fastorder_admin_gd
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: worker-01
User (CN): fastorder_admin_gd
Hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 04:56:14 UTC] USER=www-data EUID=0 PID=3788741 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-fastorder_admin_gd
[2026-02-05 04:56:14 UTC] USER=www-data EUID=0 PID=3788752 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-02-05 04:56:14 UTC] USER=www-data EUID=0 PID=3788763 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
[2026-02-05 04:56:14 UTC] USER=www-data EUID=0 PID=3788774 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-02-05 04:56:14 UTC] USER=www-data EUID=0 PID=3788786 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:56:15 UTC] USER=www-data EUID=0 PID=3788803 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:56:15 UTC] USER=www-data EUID=0 PID=3788814 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:56:15 UTC] USER=www-data EUID=0 PID=3788858 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 04:56:15 UTC] USER=www-data EUID=0 PID=3788869 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:56:15 UTC] USER=www-data EUID=0 PID=3788878 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:56:15 UTC] USER=www-data EUID=0 PID=3788889 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 04:56:15 UTC] USER=www-data EUID=0 PID=3788898 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 04:56:15 UTC] USER=www-data EUID=0 PID=3788908 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 04:56:15 UTC] USER=www-data EUID=0 PID=3788917 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 04:56:15 UTC] USER=www-data EUID=0 PID=3788926 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 04:56:15 UTC] USER=www-data EUID=0 PID=3788939 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 04:56:15 UTC] USER=www-data EUID=0 PID=3788955 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 04:56:15 UTC] USER=www-data EUID=0 PID=3788964 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:56:15 UTC] USER=www-data EUID=0 PID=3788973 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 04:56:15 UTC] USER=www-data EUID=0 PID=3788984 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 04:56:15 UTC] USER=www-data EUID=0 PID=3788993 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 04:56:15 UTC] USER=www-data EUID=0 PID=3789002 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789011 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789020 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789029 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789065 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789074 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789092 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789104 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789118 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789127 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789136 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789145 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789154 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789163 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789173 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789183 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789192 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789201 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789210 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789219 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789228 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789237 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789246 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789255 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789265 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789274 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789283 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789295 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789306 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:16 UTC] USER=www-data EUID=0 PID=3789315 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789324 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789333 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789345 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789354 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789366 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789378 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789387 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789396 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789405 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789415 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789428 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789438 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789447 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789457 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789466 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789475 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789484 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789493 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789502 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789512 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789521 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789530 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789544 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789563 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789573 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789582 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789591 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789600 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789610 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789622 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789631 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789640 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:56:17 UTC] USER=www-data EUID=0 PID=3789649 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: fastorder_admin_gd
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U fastorder_admin_gd -d postgres
🧱 Connecting via Unix socket to create role and database...
Socket: /var/run/postgresql-web-universe-main-dev-worker-01:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️ Database fastorder_web_universe_main_dev_db already exists, skipping creation
[2026-02-05 04:56:18 UTC] USER=www-data EUID=0 PID=3789710 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️ Using pg_hba.conf: /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-02-05 04:56:18 UTC] USER=www-data EUID=0 PID=3789752 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-worker-01.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[0;34m=== Pre-flight Checks ===[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;32m✓[0m AWS Secrets Manager accessible
[0;34m=== Retrieving Credentials from AWS ===[0m
ℹ️ Retrieving PostgreSQL credentials for: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️ Fetching secret: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials loaded for worker-01/fastorder_admin_gd: fastorder_admin_gd@db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_web_universe_main_dev_db
[0;32m✓[0m Credentials retrieved: fastorder_admin_gd@db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_web_universe_main_dev_db
[0;34m╔════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Test Suite (AWS Secrets MGR) ║[0m
[0;34m╚════════════════════════════════════════════╝[0m
[0;34m=== PostgreSQL Authentication Test ===[0m
[0;31m✗[0m PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-web-universe-main-dev-postgresql-worker-01.fastorder.com" (10.100.1.243), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 04:56:23 UTC] USER=www-data EUID=0 PID=3790036 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/worker-01/standby.signal
── fast setup ─────────────────────────────────────────────
NAME : web-universe-main-dev
IDENTIFIER : worker-01
PG HOST : db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432
ROLE : debezium_user
DB : fastorder_web_universe_main_dev_db
SCHEMA : web
AUTH MODE : scram (scram=password over TLS | cert=mTLS)
SUBNET ALLOW: 10.201.0.0/16
CONNECT /32 : 142.93.238.16
SSL DIR : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
DNS → 10.100.1.243
CA : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: worker-01[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-worker-01[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mGenerating new secure password...[0m
[0;34mUser debezium_user does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: debezium_user[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
✅ Secret created: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials worker-01[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: worker-01/debezium_user
✓ Retrieved password from secrets vault
password : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️ Role debezium_user exists, updating
[2026-02-05 04:56:32 UTC] USER=www-data EUID=0 PID=3790689 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️ Database fastorder_web_universe_main_dev_db already exists
[2026-02-05 04:56:32 UTC] USER=www-data EUID=0 PID=3790724 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d fastorder_web_universe_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️ Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
🔐 Creating replicator role for worker-01...
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
NAME : web-universe-main-dev
IDENTIFIER : worker-01
PG HOST : db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432
ROLE : replicator
SSL DIR : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
DNS → 10.100.1.243
CA : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
🔑 Generating new secure replicator password...
💾 Storing replicator password in AWS Secrets Manager...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/db/web/universe/main/dev/postgresql/replicator-TNLuej",
"Name": "fastorder/db/web/universe/main/dev/postgresql/replicator",
"VersionId": "9ffa8590-1159-43c9-a3f9-0e659476859a"
}
✅ Password stored in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/replicator
ℹ️ Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE: Creating role: replicator with password
SET
CREATE ROLE
✅ Replicator role ensured with password authentication.
ℹ️ Password stored in: AWS Secrets Manager
Secret name: fastorder/db/web/universe/main/dev/postgresql/replicator
🔄 MIGRATION PATH: Password → Certificate Authentication
Current: SCRAM-SHA-256 password auth (production-ready)
Future: Certificate-based auth (requires CA automation)
To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
and configure standby to use SSL certificates instead of password
🎉 Done.
✅ Replicator role created for worker-01
[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=752a811e-1a02-4989-b8c6-1afd6a8f211b)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
ℹ️ Service-specific setup (web) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)
🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=web
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
🔍 DEBUG_CHECKPOINT_03: No specific folder for web, using default
[DEBUG] Tracking substep start: steps/01-install/steps/default (RUN_UUID=752a811e-1a02-4989-b8c6-1afd6a8f211b)
[INFO] 🔸 Service: web (using default contracts schema)
🔍 DEBUG_CHECKPOINT_04: Executing default: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/run.sh
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 🟢 Starting default contracts schema provisioning for SERVICE=web
[INFO] Environment: web-universe-main-dev
[INFO] Schema: web (contracts tables)
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Skipping Schema Setup on worker-01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ℹ️ Schema setup only runs on coordinator
ℹ️ This is a worker-01 node - schemas replicate automatically
✅ Nothing to do on this node
[0;32m✓[0m ✅ Worker worker-01 setup completed
Skipping standbys (PG_WORKERS_STANDBY_NUM=0)
[0;32m✓[0m ✅ PostgreSQL installation completed
[0;34m[INFO][0m Discovering additional setup steps...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 02-pg-bouncer.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up PgBouncer connection pooling...
[2026-02-05 04:56:42 UTC] USER=www-data EUID=0 PID=3791125 ACTION=fsop ARGS=rm -f /tmp/pgbouncer-ip.service /tmp/pgbouncer.service
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;32m✓ [SECRETS][0m Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[0;34m[SECRETS][0m Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[0;34m[SECRETS][0m Search (build_es_secret_name, get/set_es_credentials_to_vault)
[0;34m[SECRETS][0m Backups (build_backup_path)
[0;34m[SECRETS][0m Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[0;34m[INFO][0m Checking for existing PgBouncer application environment in topology …
[0;34m[INFO][0m PgBouncer application not found in topology, creating new environment …
[INFO] 🎯 Custom Environment Creation (Example Wrapper)
[INFO] 📁 Orchestrator Library: /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] 💾 State Directory: /opt/fastorder/bash/scripts/env_app_setup/state
[INFO] 🚀 Calling centralized orchestrator: fo-env create-app
[INFO] 📋 Arguments: --service web --zone universe --branch main --env dev --domain db-web-universe-main-dev-postgresql-bouncer --app pgbouncer
[INFO] Creating application-specific environment configuration
[INFO] Environment ID: web-universe-main-dev
[INFO] Application: pgbouncer
[INFO] Base environment web-universe-main-dev already exists
[INFO] Allocated pgbouncer IP: 10.100.1.244
[INFO] Generated domain: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO] Configuring network interface for pgbouncer IP: 10.100.1.244
[2026-02-05 04:56:44 UTC] USER=www-data EUID=0 PID=3791700 ACTION=passthru ARGS=ip addr add 10.100.1.244/32 dev eth0 label eth0:244
[ OK ] Configured pgbouncer IP 10.100.1.244 on interface eth0:244
[INFO] Creating systemd service for pgbouncer IP persistence...
[2026-02-05 04:56:44 UTC] USER=www-data EUID=0 PID=3791733 ACTION=passthru ARGS=systemctl daemon-reload
[ OK ] pgbouncer IP will persist across reboots
[INFO] Updating topology with application-specific configuration...
[ OK ] Topology updated with application-specific configuration
[INFO] Binding pgbouncer IP to domain: 10.100.1.244 -> db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[ OK ] Successfully bound db-web-universe-main-dev-postgresql-bouncer.fastorder.com to 10.100.1.244
[ OK ] Domain correctly mapped
[ OK ] Application environment created successfully!
[INFO]
[INFO] Application Details:
[INFO] Environment ID: web-universe-main-dev
[INFO] Application: pgbouncer
[INFO] IP: 10.100.1.244
[INFO] Domain: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO]
[INFO] To use this application:
[INFO] source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO] init_environment pgbouncer
[INFO] echo $VM_IP # Returns: 10.100.1.244
[ OK ] 🎉 Environment creation completed successfully!
[INFO] 📋 What happened:
[INFO] ✅ Called centralized orchestrator at /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] ✅ All topology.json management handled centrally
[INFO] ✅ Application-specific IP and domain configured
[INFO] ✅ Network interface configured and made persistent
[INFO] ✅ Domain binding added to /etc/hosts (if not skipped)
[INFO] 🔧 To use the centralized orchestrator directly:
[INFO] # Add orchestrator to PATH
[INFO] export PATH="/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/bin:$PATH"
[INFO] # Then call directly
[INFO] fo-env create-app --service auth --zone uae --env dev --app redis
[INFO] 📚 For more orchestrator commands:
[INFO] fo-env --help
[0;32m[OK][0m Created new PgBouncer environment:
[0;34m[INFO][0m IP: 10.100.1.244
[0;34m[INFO][0m FQDN: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[0;34m[INFO][0m Domain: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[0;34m[INFO][0m Final verification of /etc/hosts entry for db-web-universe-main-dev-postgresql-bouncer.fastorder.com …
[0;32m[OK][0m /etc/hosts correctly maps db-web-universe-main-dev-postgresql-bouncer.fastorder.com to 10.100.1.244
[1;33m[WARN][0m IP 10.100.1.244 is already bound to other interface(s):
inet 10.100.1.244/32 scope global eth0:244
[0;34m[INFO][0m Attempting to also bind 10.100.1.244 to lo:pgbouncer ...
[2026-02-05 04:56:46 UTC] USER=www-data EUID=0 PID=3791906 ACTION=passthru ARGS=ip addr add 10.100.1.244/32 dev lo label lo:pgbouncer
[0;32m[OK][0m Successfully bound 10.100.1.244 to lo:pgbouncer
[2026-02-05 04:56:46 UTC] USER=www-data EUID=0 PID=3791924 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 04:56:47 UTC] USER=www-data EUID=0 PID=3792067 ACTION=passthru ARGS=systemctl restart pgbouncer-ip@web-universe-main-dev.service
[2026-02-05 04:56:47 UTC] USER=www-data EUID=0 PID=3792077 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer-ip@web-universe-main-dev.service
[1;33m[WARN][0m pgbouncer-ip@web-universe-main-dev.service is not active
[1;33m[WARN][0m Check status: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@web-universe-main-dev.service
[2026-02-05 04:56:47 UTC] USER=www-data EUID=0 PID=3792107 ACTION=fsop ARGS=mkdir -p /etc/pgbouncer/web-universe-main-dev
[2026-02-05 04:56:47 UTC] USER=www-data EUID=0 PID=3792116 ACTION=fsop ARGS=mkdir -p /run/pgbouncer/web-universe-main-dev
[2026-02-05 04:56:47 UTC] USER=www-data EUID=0 PID=3792125 ACTION=fsop ARGS=mkdir -p /var/log/pgbouncer/web-universe-main-dev
[2026-02-05 04:56:47 UTC] USER=www-data EUID=0 PID=3792134 ACTION=fsop ARGS=chmod 750 /etc/pgbouncer/web-universe-main-dev
[2026-02-05 04:56:47 UTC] USER=www-data EUID=0 PID=3792143 ACTION=fsop ARGS=chmod 750 /run/pgbouncer/web-universe-main-dev
[2026-02-05 04:56:47 UTC] USER=www-data EUID=0 PID=3792152 ACTION=fsop ARGS=chmod 750 /var/log/pgbouncer/web-universe-main-dev
[2026-02-05 04:56:47 UTC] USER=www-data EUID=0 PID=3792161 ACTION=fsop ARGS=chown root:postgres /etc/pgbouncer/web-universe-main-dev
[2026-02-05 04:56:47 UTC] USER=www-data EUID=0 PID=3792170 ACTION=fsop ARGS=chown postgres:postgres /run/pgbouncer/web-universe-main-dev
[2026-02-05 04:56:47 UTC] USER=www-data EUID=0 PID=3792179 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbouncer/web-universe-main-dev
[0;34m[INFO][0m Generating PgBouncer TLS certificate via existing server.sh (IDENTIFIER=pgbouncer) …
[0;34m[INFO][0m Calling: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/ssl/server.sh pgbouncer
[0;34m[INFO][0m With: OVERRIDE_CN=db-web-universe-main-dev-postgresql-bouncer.fastorder.com, OVERRIDE_IP=10.100.1.244
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe (FastOrder Universe)
Branch: main
Env: dev
Node: pgbouncer
Primary CN: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
Alt CN: web-universe-main-dev.fastorder.com
VM IP: 10.100.1.244
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 04:56:48 UTC] USER=www-data EUID=0 PID=3792227 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer and /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:48 UTC] USER=www-data EUID=0 PID=3792236 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
🔐 Generating 4096-bit private key...
[2026-02-05 04:56:48 UTC] USER=www-data EUID=0 PID=3792246 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-3792186
[2026-02-05 04:56:48 UTC] USER=www-data EUID=0 PID=3792259 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-3792186/ra_root.crt
[2026-02-05 04:56:48 UTC] USER=www-data EUID=0 PID=3792268 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-3792186/ra_root.key
[2026-02-05 04:56:48 UTC] USER=www-data EUID=0 PID=3792277 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3792186/ra_root.crt
[2026-02-05 04:56:48 UTC] USER=www-data EUID=0 PID=3792286 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3792186/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[2026-02-05 04:56:51 UTC] USER=www-data EUID=0 PID=3792390 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3792186/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
[2026-02-05 04:56:51 UTC] USER=www-data EUID=0 PID=3792399 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3792186/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
[2026-02-05 04:56:51 UTC] USER=www-data EUID=0 PID=3792408 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
📋 Setting up CA certificate...
[2026-02-05 04:56:51 UTC] USER=www-data EUID=0 PID=3792417 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3792186/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 04:56:51 UTC] USER=www-data EUID=0 PID=3792441 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 04:56:51 UTC] USER=www-data EUID=0 PID=3792452 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 04:56:51 UTC] USER=www-data EUID=0 PID=3792461 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 04:56:51 UTC] USER=www-data EUID=0 PID=3792473 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
[2026-02-05 04:56:51 UTC] USER=www-data EUID=0 PID=3792484 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
[2026-02-05 04:56:52 UTC] USER=www-data EUID=0 PID=3792503 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
[2026-02-05 04:56:52 UTC] USER=www-data EUID=0 PID=3792513 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:52 UTC] USER=www-data EUID=0 PID=3792523 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
🔍 Verifying certificate...
Certificate details:
Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-bouncer.fastorder.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
--
X509v3 Subject Alternative Name:
DNS:db-web-universe-main-dev-postgresql-bouncer.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-bouncer.fastorder.com, DNS:db-web-universe-main-dev-postgresql-bouncer, DNS:localhost, IP Address:10.100.1.244, IP Address:127.0.0.1
X509v3 Subject Key Identifier:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Node: pgbouncer
Primary CN: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt'
2. Restart PgBouncer:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@web-universe-main-dev.service
3. Test SSL connection:
psql "host=db-web-universe-main-dev-postgresql-bouncer.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m mTLS server certificate present: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
[0;34m[INFO][0m Generating pgbouncer_admin client certificates...
[0;34m[INFO][0m ⏳ This may take 30-60 seconds...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: pgbouncer_admin
Identifier: pgbouncer
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: pgbouncer
User (CN): pgbouncer_admin
Hostname: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 04:56:52 UTC] USER=www-data EUID=0 PID=3792592 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-pgbouncer-pgbouncer_admin
[2026-02-05 04:56:52 UTC] USER=www-data EUID=0 PID=3792601 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-02-05 04:56:52 UTC] USER=www-data EUID=0 PID=3792610 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
[2026-02-05 04:56:52 UTC] USER=www-data EUID=0 PID=3792619 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-02-05 04:56:52 UTC] USER=www-data EUID=0 PID=3792630 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = pgbouncer_admin
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792655 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792666 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792675 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792684 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792695 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792704 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792721 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792730 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792748 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792757 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792766 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792775 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792785 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792795 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792804 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792818 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792829 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792860 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:53 UTC] USER=www-data EUID=0 PID=3792869 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3792878 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3792905 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3792914 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3792923 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3792932 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3792941 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3792950 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3792959 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3792969 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3792979 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3792989 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3793002 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3793014 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3793029 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3793040 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3793058 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3793076 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3793085 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3793094 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3793104 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3793123 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3793133 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3793151 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:54 UTC] USER=www-data EUID=0 PID=3793161 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793182 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793191 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793209 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793219 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793229 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793239 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793249 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793258 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793267 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793277 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793287 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793298 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793307 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793316 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793328 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793339 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793358 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793388 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:56:55 UTC] USER=www-data EUID=0 PID=3793413 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793431 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
✅ Symlinked client-key.pem
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793449 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793458 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793467 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793476 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: pgbouncer_admin
Node: pgbouncer
FQDN: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-bouncer.fastorder.com -U pgbouncer_admin -d postgres
[0;32m[OK][0m mTLS client certificate present: /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[0;34m[INFO][0m Creating symlinks to canonical certificates in /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend...
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793492 ACTION=fsop ARGS=mkdir -p /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793501 ACTION=fsop ARGS=mkdir -p /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793510 ACTION=fsop ARGS=ln -sf /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793519 ACTION=fsop ARGS=ln -sf /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793529 ACTION=fsop ARGS=ln -sf /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt
[0;34m[INFO][0m Creating coordinator CA symlink for PostgreSQL server verification...
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793538 ACTION=fsop ARGS=ln -sf /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m Verifying canonical certificate permissions...
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793549 ACTION=fsop ARGS=chmod 644 /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793560 ACTION=fsop ARGS=chmod 640 /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793569 ACTION=fsop ARGS=chmod 644 /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793578 ACTION=fsop ARGS=chown root:www-data /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[0;32m[OK][0m Backend certificate symlinks created in /etc/ssl
[0;32m[OK][0m Coordinator CA symlink created for server verification
[0;34m[INFO][0m Creating symlinks in /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer for monitoring access...
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793589 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793598 ACTION=fsop ARGS=ln -sf /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
/bin/ln: '/home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt' and '/etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt' are the same file
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793607 ACTION=fsop ARGS=ln -sf /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[0;32m[OK][0m Monitoring certificate symlinks created (or already exist)
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793616 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793625 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793634 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793643 ACTION=fsop ARGS=test -r /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m PgBouncer will use PostgreSQL coordinator CA: /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;32m[OK][0m PostgreSQL coordinator at db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432 is reachable
[0;34m[INFO][0m Dumping SCRAM secrets from coordinator for PgBouncer auth_file …
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793664 ACTION=fsop ARGS=cp /tmp/tmp.gUPXnoa47c /etc/pgbouncer/web-universe-main-dev/userlist.txt
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793677 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/userlist.txt
[2026-02-05 04:56:56 UTC] USER=www-data EUID=0 PID=3793709 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;32m[OK][0m Auth file written: /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;34m[INFO][0m Generated new password for pgbouncer_admin
[0;34m[INFO][0m Ensuring PgBouncer admin role 'pgbouncer_admin' exists in Postgres (coordinator) …
[0;32m[OK][0m Role pgbouncer_admin created/updated successfully
[0;34m[SECRETS][0m Setting credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/pgbouncer_admin
[0;32m✓ [SECRETS][0m Credentials created in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/pgbouncer_admin
[0;34m[INFO][0m ✅ PgBouncer admin password stored in centralized secrets vault
[0;34m[INFO][0m Re-fetching SCRAM secrets after role creation to ensure pgbouncer_admin is included …
[2026-02-05 04:57:03 UTC] USER=www-data EUID=0 PID=3794076 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/userlist.txt
[2026-02-05 04:57:03 UTC] USER=www-data EUID=0 PID=3794085 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;32m[OK][0m Auth file updated with pgbouncer_admin SCRAM hash
[0;34m[INFO][0m Auth file contains [2026-02-05 04:57:03 UTC] USER=www-data EUID=0 PID=3794095 ACTION=passthru ARGS=bash -c wc -l < '/etc/pgbouncer/web-universe-main-dev/userlist.txt'
4 user(s)
[0;32m[OK][0m Admin 'pgbouncer_admin' password generated and saved
[0;34m[INFO][0m Configuring PostgreSQL to prevent Citus metadata sync hangs...
ALTER ROLE
[0;32m[OK][0m Disabled Citus metadata sync for pgbouncer_admin
[0;34m[INFO][0m Verifying application database fastorder_web_universe_main_dev_db exists...
[0;32m[OK][0m ✓ Database fastorder_web_universe_main_dev_db exists
[0;34m[INFO][0m Granting permissions to pgbouncer_admin on fastorder_web_universe_main_dev_db...
GRANT
[0;32m[OK][0m ✓ Granted CONNECT on fastorder_web_universe_main_dev_db to pgbouncer_admin
GRANT
[0;32m[OK][0m ✓ Granted USAGE on schema public to pgbouncer_admin
GRANT
[0;32m[OK][0m ✓ Granted SELECT on all tables to pgbouncer_admin
ALTER DATABASE
[0;32m[OK][0m Set synchronous_commit=local for fastorder_web_universe_main_dev_db
[0;34m[INFO][0m Ensuring pg_hba.conf entry for pgbouncer_admin …
[0;34m[INFO][0m Adding pg_hba.conf entries for pgbouncer_admin with cert auth …
[0;32m[OK][0m pg_hba.conf updated and PostgreSQL configuration reloaded
[2026-02-05 04:57:04 UTC] USER=unknown EUID=33 PID=3794142 ACTION=-u ARGS=postgres bash
ERROR: Invalid or unauthorized action: -u
[1;33m[WARN][0m pg_hba.conf entry may not have loaded correctly
[0;34m[INFO][0m Writing /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini …
[2026-02-05 04:57:05 UTC] USER=www-data EUID=0 PID=3794218 ACTION=fsop ARGS=cp /tmp/tmp.sVBrdG3sr4 /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
[2026-02-05 04:57:05 UTC] USER=www-data EUID=0 PID=3794227 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
[2026-02-05 04:57:05 UTC] USER=www-data EUID=0 PID=3794245 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbouncer/web-universe-main-dev /run/pgbouncer/web-universe-main-dev /var/log/pgbouncer/web-universe-main-dev
[2026-02-05 04:57:05 UTC] USER=www-data EUID=0 PID=3794254 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;32m[OK][0m pgbouncer.ini ready
[0;34m[INFO][0m Verifying TLS settings in pgbouncer.ini:
[2026-02-05 04:57:05 UTC] USER=www-data EUID=0 PID=3794264 ACTION=fsop ARGS=grep -E (client_tls_sslmode|server_tls) /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
client_tls_sslmode = verify-full
server_tls_sslmode = verify-full
server_tls_ca_file = /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
server_tls_cert_file = /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
server_tls_key_file = /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;34m[INFO][0m Verifying PgBouncer server certificate files:
[2026-02-05 04:57:05 UTC] USER=www-data EUID=0 PID=3794273 ACTION=fsop ARGS=test -r /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[0;32m[OK][0m Server cert readable by postgres: /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-02-05 04:57:05 UTC] USER=www-data EUID=0 PID=3794284 ACTION=fsop ARGS=test -r /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;32m[OK][0m Server key readable by postgres: /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;34m[INFO][0m Verifying coordinator CA certificate:
[2026-02-05 04:57:05 UTC] USER=www-data EUID=0 PID=3794293 ACTION=fsop ARGS=test -r /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;32m[OK][0m Coordinator CA readable by postgres: /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m Preflight: stopping any conflicting PgBouncer on 6432 …
[2026-02-05 04:57:05 UTC] USER=www-data EUID=0 PID=3794304 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer.service
[2026-02-05 04:57:05 UTC] USER=www-data EUID=0 PID=3794313 ACTION=passthru ARGS=systemctl stop pgbouncer@web-universe-main-dev.service
Failed to stop pgbouncer@web-universe-main-dev.service: Unit pgbouncer@web-universe-main-dev.service not loaded.
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json?all=1": dial unix /var/run/docker.sock: connect: permission denied
[2026-02-05 04:57:08 UTC] USER=www-data EUID=0 PID=3794424 ACTION=passthru ARGS=systemctl daemon-reload
[0;32m[OK][0m systemd unit installed: pgbouncer@web-universe-main-dev.service
[0;34m[INFO][0m Running pre-flight IP conflict check for 10.100.1.244:6432 …
[1;33m[WARN][0m IP conflict checker not found at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/check-ip-conflicts.sh
[1;33m[WARN][0m Skipping pre-flight check - conflicts may occur
[0;34m[INFO][0m Starting PgBouncer (web-universe-main-dev) …
[2026-02-05 04:57:09 UTC] USER=www-data EUID=0 PID=3794519 ACTION=passthru ARGS=systemctl restart pgbouncer@web-universe-main-dev.service
[2026-02-05 04:57:09 UTC] USER=www-data EUID=0 PID=3794532 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer@web-universe-main-dev.service
[0;32m[OK][0m Service ACTIVE
[0;34m[INFO][0m Verifying auth_file before probing …
[0;34m[INFO][0m Auth file contains 4 user(s)
[1;33m[WARN][0m Auth file does NOT contain pgbouncer_admin entry - authentication will fail
[0;34m[INFO][0m Probing admin console via SSL (psql to database 'pgbouncer') …
[0;34m[INFO][0m Retrieved password from vault for admin console probe
[1;33m[WARN][0m Admin console probe failed (see error below)
psql: error: connection to server at "10.100.1.244", port 6432 failed: root certificate file "/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
[1;33m[WARN][0m Troubleshooting:
[1;33m[WARN][0m 1. Check auth_file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/web-universe-main-dev/userlist.txt
[1;33m[WARN][0m 2. Test with: PGPASSWORD='WO0D2C0d7ZbIdk65D10y9TaD' psql -h 10.100.1.244 -p 6432 -U pgbouncer_admin -d pgbouncer
[1;33m[WARN][0m 3. Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@web-universe-main-dev.service -n 50
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Running Comprehensive PgBouncer Verification Tests
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Password extracted: WO0D2C0d7Z... (using postgres user certificates)
[0;34m[INFO][0m Test 1/7: Admin Console - SHOW POOLS
[1;33m[WARN][0m ✗ SHOW POOLS: FAILED
[1;33m[WARN][0m Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@web-universe-main-dev.service -n 50
[0;34m[INFO][0m Test 2/7: Admin Console - SHOW VERSION
[1;33m[WARN][0m ✗ SHOW VERSION: FAILED
[0;34m[INFO][0m Test 3/7: Admin Console - SHOW STATS
[1;33m[WARN][0m ✗ SHOW STATS: FAILED
[0;34m[INFO][0m Test 4/7: Admin Console - SHOW DATABASES
[1;33m[WARN][0m ✗ SHOW DATABASES: FAILED
[0;34m[INFO][0m Test 5/7: Admin Console - SHOW CONFIG
[1;33m[WARN][0m ✗ SHOW CONFIG: FAILED
psql "host=db-web-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_web_universe_main_dev_db user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD connect_timeout=5 sslmode=verify-full sslrootcert=/home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt sslcert=/home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt sslkey=/home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key" --no-psqlrc -Atc 'SELECT version();'
[0;34m[INFO][0m Test 6/7: Application Database - SELECT version()
[1;33m[WARN][0m ✗ Application database query: FAILED (timeout or connection issue)
[1;33m[WARN][0m If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[0;34m[INFO][0m Test 7/8: Application Database - Connection Details
[1;33m[WARN][0m ✗ Connection details: FAILED (timeout or connection issue)
[1;33m[WARN][0m If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[0;34m[INFO][0m Test 8/8: End-to-End Application Routing - Pool Verification
[0;34m[INFO][0m Running actual queries through PgBouncer to verify routing and pooling...
[1;33m[WARN][0m ✗ End-to-end routing verification: FAILED - All 3 queries failed
[1;33m[WARN][0m If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[1;33m[WARN][0m Otherwise check if database fastorder_web_universe_main_dev_db exists and user pgbouncer_admin has permissions
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verification Complete - Tests 1-5 PASSED (Admin console verified)
[1;33m[WARN][0m Tests 6-8 FAILED - Application database not accessible
[1;33m[WARN][0m This is expected if Citus is not set up yet
[1;33m[WARN][0m Run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m PgBouncer is up for web-universe-main-dev
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Connection Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Password stored in: AWS Secrets Manager (fastorder/db/web/ksa/main/dev/postgresqlweb/universe/main/dev/coordinator-pgbouncer_admin)
Current password: WO0D2C0d7ZbIdk65D10y9TaD
1. Admin Console (using IP address to avoid DNS/SSL issues):
psql "host=10.100.1.244 port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD sslmode=verify-full sslrootcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"
2. Admin Console (using hostname):
psql "host=db-web-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD sslmode=verify-full sslrootcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"
3. Application Database:
psql "host=db-web-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_web_universe_main_dev_db sslkey=/etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD sslmode=verify-full sslrootcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"
4. Using .pgpass file:
echo "db-web-universe-main-dev-postgresql-bouncer.fastorder.com:6432:*:pgbouncer_admin:WO0D2C0d7ZbIdk65D10y9TaD" >> ~/.pgpass
chmod 600 ~/.pgpass
psql -h db-web-universe-main-dev-postgresql-bouncer.fastorder.com -p 6432 -U pgbouncer_admin -d fastorder_web_universe_main_dev_db
5. Retrieve password from vault:
source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
PGPASSWORD="$(get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password')" \
psql -h 10.100.1.244 -p 6432 -U pgbouncer_admin -d pgbouncer -c "SHOW POOLS;"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Architecture
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
• Default db 'fastorder_web_universe_main_dev_db' → Citus coordinator (db-web-universe-main-dev-postgresql-coordinator.fastorder.com)
• Worker access: 'fastorder_web_universe_main_dev_db_worker_1', 'fastorder_web_universe_main_dev_db_worker_2', … (if exist)
• Client TLS: require (password auth) / verify-full (mTLS with certs)
• Server TLS: verify-full (PgBouncer validates PostgreSQL certs)
• Auth: SCRAM-SHA-256 via /etc/pgbouncer/web-universe-main-dev/userlist.txt
• Pool mode: transaction (stateless connections)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Management
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Service Status:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer@web-universe-main-dev.service
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@web-universe-main-dev.service
Logs:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@web-universe-main-dev.service -f
/usr/local/bin/fastorder-provisioning-wrapper.sh tail -f /var/log/pgbouncer/web-universe-main-dev/pgbouncer.log
Reload Config:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@web-universe-main-dev.service
Restart:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@web-universe-main-dev.service
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Files
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Config: /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
Auth file: /etc/pgbouncer/web-universe-main-dev/userlist.txt
Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
Server key: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
CA cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt
PG CA: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
Logs: /var/log/pgbouncer/web-universe-main-dev/pgbouncer.log
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Troubleshooting
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
If "SASL authentication failed":
1. Check auth file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/web-universe-main-dev/userlist.txt
2. Verify pgbouncer_admin is present with SCRAM hash
3. Get password from vault:
source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password'
4. Reload PgBouncer: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@web-universe-main-dev.service
If "no pg_hba.conf entry":
1. Check pg_hba.conf on coordinator
2. Add rule: hostssl all pgbouncer_admin 10.100.1.244/32 cert clientcert=verify-full
3. Reload PostgreSQL
To add users to PgBouncer:
1. Create user in PostgreSQL with password
2. Re-run SCRAM dump:
psql "host=db-web-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 dbname=postgres user=postgres \
sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt \
sslcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt sslkey=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key" \
-Atc "SELECT '\"' || rolname || '\" \"' || rolpassword || '\"' \
FROM pg_authid WHERE rolpassword LIKE 'SCRAM-SHA-256%' \
AND rolcanlogin ORDER BY rolname;" | command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop tee /etc/pgbouncer/web-universe-main-dev/userlist.txt
3. Reload: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@web-universe-main-dev.service
[0;34m[INFO][0m Registering PgBouncer node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: PgBouncer
[INFO] Identifier: web-universe-main-dev-pgbouncer
[INFO] Identifier Parent: postgresql
[INFO] IP: 10.100.1.244
[INFO] Port: 6432
[INFO] FQDN: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: fdc70c5f-615d-432f-8161-a7acd56ea9ed
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[0;32m[OK][0m PgBouncer node registered to observability API
[0;32m✓[0m ✅ PgBouncer setup completed
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 03-citus-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m CITUS DISTRIBUTED CLUSTER SETUP
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Phase 1: Installing Citus extension on workers...
[0;34m[INFO][0m Phase 2: Setting up coordinator and registering workers...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 📦 PHASE 1: Installing Citus extension on 1 worker(s)...
[0;34m[INFO][0m → Worker 1/1: Installing Citus on worker-01...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS CLUSTER SETUP
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m 🔧 Setting up Citus Worker...
[0;34m[INFO][0m Temporarily disabling synchronous replication for extension installation...
t
[0;34m[INFO][0m Installing Citus extension on worker...
[0;32m[OK][0m Citus extension installed on worker
[0;34m[INFO][0m Restoring synchronous replication settings...
t
[0;34m[INFO][0m Worker Citus extension installed - registration will happen when coordinator setup runs
[0;32m[OK][0m Citus setup complete for worker-01
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m✓[0m ✅ Citus extension installed on worker-01
[0;32m✓[0m ✅ Phase 1 Complete: All 1 workers have Citus extension installed
[0;34m[INFO][0m 🔧 PHASE 2: Setting up Citus coordinator and registering workers...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS CLUSTER SETUP
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m 🔧 Setting up Citus Coordinator...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m DIAGNOSTIC: Configuration Variables
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m PG_WORKERS_NUM: 1
[0;34m[INFO][0m ENV_ID: web-universe-main-dev
[0;34m[INFO][0m DOMAIN: fastorder.com
[0;34m[INFO][0m PORT: 5432
[0;34m[INFO][0m SOCKET_DIR: /var/run/postgresql-web-universe-main-dev-coordinator
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Ensuring postgres client certificates exist for coordinator...
[0;32m[OK][0m Postgres client certificates already exist for coordinator
[0;34m[INFO][0m Adding citus_cert_map to coordinator pg_ident.conf...
[0;32m[OK][0m pg_ident.conf updated for coordinator
[0;34m[INFO][0m Installing Citus extension on coordinator...
[0;32m[OK][0m Citus extension installed on coordinator (postgres database)
[0;34m[INFO][0m Installing Citus extension on application database: fastorder_web_universe_main_dev_db...
[0;32m[OK][0m Citus extension installed on application database: fastorder_web_universe_main_dev_db
[0;34m[INFO][0m Configuring Citus SSL connection parameters...
[2026-02-05 04:57:22 UTC] USER=www-data EUID=0 PID=3795259 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m ✅ Citus SSL connection parameters configured: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[1;33m[WARN][0m Node not identified as coordinator, initializing...
[0;34m[INFO][0m Checking coordinator configuration...
[0;34m[INFO][0m Persisting citus.local_hostname to postgresql.conf...
[2026-02-05 04:57:24 UTC] USER=www-data EUID=0 PID=3795367 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/web-universe-main-dev/coordinator/postgresql.conf
[2026-02-05 04:57:24 UTC] USER=www-data EUID=0 PID=3795388 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m ✅ citus.local_hostname persisted to config and reloaded
[0;34m[INFO][0m Configuring coordinator hostname in postgres database: db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432
[0;32m[OK][0m ✅ Coordinator hostname set to db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432 in postgres database
[0;34m[INFO][0m Checking coordinator configuration in application database: fastorder_web_universe_main_dev_db...
[1;33m[WARN][0m ⚠️ Coordinator registered as 'localhost' in application database, fixing...
[0;34m[INFO][0m Configuring coordinator hostname in application database: db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432
[0;32m[OK][0m ✅ Coordinator hostname set to db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432 in application database
[0;34m[INFO][0m Validating coordinator configuration before worker registration...
[0;32m[OK][0m ✅ Coordinator hostname validated: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[0;32m[OK][0m ✅ citus_tables view is accessible
[0;34m[INFO][0m Checking coordinator self-registration...
[0;32m[OK][0m ✅ Coordinator is already self-registered
[0;34m[INFO][0m Configuring coordinator shard placement policy...
[0;32m[OK][0m ✅ Coordinator already configured in postgres database (shouldhaveshards = false)
[1;33m[WARN][0m ⚠️ Coordinator has 66 shards in fastorder_web_universe_main_dev_db - cannot set shouldhaveshards=false
[1;33m[WARN][0m You must rebalance shards to workers first, then run this setup again
[1;33m[WARN][0m Skipping shouldhaveshards configuration for application database
[0;34m[INFO][0m Registering 1 worker(s) to Citus cluster...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m PRE-FLIGHT: Checking worker availability...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Checking worker worker-01...
[0;34m[INFO][0m FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[0;32m[OK][0m ✅ Worker worker-01 is reachable via SSL
[0;32m[OK][0m All workers are reachable - proceeding with registration
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Adding Citus worker: db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Adding citus_cert_map to worker-01 pg_ident.conf...
[0;32m[OK][0m pg_ident.conf updated for worker-01
[0;34m[INFO][0m Configuring worker worker-01 HBA for coordinator (10.100.1.119) access...
[0;32m[OK][0m Worker worker-01 HBA configured for coordinator (10.100.1.119)
[0;34m[INFO][0m Adding replication rules for 3 standby(s)...
[0;32m[OK][0m Replication rules added for worker-01
[0;34m[INFO][0m Reloading worker worker-01 to apply HBA changes...
[2026-02-05 04:57:30 UTC] USER=www-data EUID=0 PID=3795910 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-worker-01.service
[0;34m[INFO][0m Configuring coordinator HBA for worker worker-01 (10.100.1.243) access...
[0;32m[OK][0m Coordinator HBA configured for worker worker-01 (10.100.1.243)
[0;34m[INFO][0m Reloading coordinator to apply HBA changes...
[2026-02-05 04:57:30 UTC] USER=www-data EUID=0 PID=3795946 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
[0;34m[INFO][0m Ensuring postgres client certificates exist for worker-01...
[0;32m[OK][0m Postgres client certificates already exist for worker-01
[0;34m[INFO][0m Configuring citus.node_conninfo on worker-01...
[2026-02-05 04:57:30 UTC] USER=www-data EUID=0 PID=3795971 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-worker-01.service
[0;32m[OK][0m citus.node_conninfo configured on worker-01
[0;34m[INFO][0m Temporarily relaxing sync-rep on worker worker-01...
t
[0;32m[OK][0m Worker worker-01 sync-rep relaxed (was: sync_commit=on)
[0;34m[INFO][0m Ensuring Citus extension on worker databases...
CREATE EXTENSION
CREATE EXTENSION
[0;34m[INFO][0m Running citus_add_node with 180s timeout...
NOTICE: shards are still on the coordinator after adding the new node
HINT: Use SELECT rebalance_table_shards(); to balance shards data between workers and coordinator or SELECT citus_drain_node('db-web-universe-main-dev-postgresql-coordinator.fastorder.com',5432); to permanently move shards away from the coordinator.
2
[0;34m[INFO][0m Restoring worker worker-01 sync-rep settings...
t
[0;32m[OK][0m Worker worker-01 sync-rep restored
[0;32m[OK][0m ✅ Worker db-web-universe-main-dev-postgresql-worker-01.fastorder.com successfully added to Citus cluster
[0;34m[INFO][0m Node ID: 2
[0;34m[INFO][0m Registered in: postgres, fastorder_web_universe_main_dev_db
[0;32m[OK][0m Worker worker-01 registration successful
[0;34m[INFO][0m Configuring worker worker-01 shard placement policy...
[0;32m[OK][0m ✅ Worker worker-01 configured to hold shards in all databases
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m POST-REGISTRATION: Verifying cluster state...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Expected workers: 1
[0;34m[INFO][0m Registered workers: 1
[0;32m[OK][0m ✅ All 1 workers successfully registered!
[0;34m[INFO][0m Citus cluster configuration:
db-web-universe-main-dev-postgresql-coordinator.fastorder.com 5432 0 t primary f
db-web-universe-main-dev-postgresql-worker-01.fastorder.com 5432 1 t primary t
[0;34m[INFO][0m Note: groupid=0 is the coordinator, groupid>0 are workers
[0;34m[INFO][0m shouldhaveshards: false=query router only, true=holds data shards
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m FINAL VALIDATION: Verifying configuration persistence...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 04:57:35 UTC] USER=www-data EUID=0 PID=3796258 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/web-universe-main-dev/coordinator/postgresql.conf
[0;32m[OK][0m ✅ citus.local_hostname persisted in postgresql.conf
[0;32m[OK][0m ✅ All 1 worker(s) successfully registered and verified
[0;32m[OK][0m ✅ All validation checks passed
[0;32m[OK][0m Citus coordinator setup complete
[0;32m[OK][0m Citus setup complete for coordinator
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m✓[0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✓[0m ✅ CITUS CLUSTER SETUP COMPLETED SUCCESSFULLY
[0;32m✓[0m Coordinator: Ready and accepting connections
[0;32m✓[0m Workers registered: 1
[0;32m✓[0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 05-backup-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up coordinator backup...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for web-universe-main-dev...
[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO] Version: pgBackRest 2.56.0
[INFO] 2️⃣ Creating backup directories...
[2026-02-05 04:57:37 UTC] USER=www-data EUID=0 PID=3796367 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 04:57:37 UTC] USER=www-data EUID=0 PID=3796376 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/web-universe-main-dev
[2026-02-05 04:57:37 UTC] USER=www-data EUID=0 PID=3796390 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-02-05 04:57:37 UTC] USER=www-data EUID=0 PID=3796413 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-02-05 04:57:37 UTC] USER=www-data EUID=0 PID=3796422 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-02-05 04:57:45 UTC] USER=www-data EUID=0 PID=3796627 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-02-05 04:57:45 UTC] USER=www-data EUID=0 PID=3796642 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-02-05 04:57:45 UTC] USER=www-data EUID=0 PID=3796667 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-02-05 04:57:45 UTC] USER=www-data EUID=0 PID=3796712 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO] ✅ Backup directories created
[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Generating new cipher key and saving to /etc/pgbackrest/.cipher-key-web-universe-main-dev
[2026-02-05 04:57:45 UTC] USER=www-data EUID=0 PID=3796744 ACTION=fsop ARGS=chmod 600 /etc/pgbackrest/.cipher-key-web-universe-main-dev
[2026-02-05 04:57:46 UTC] USER=www-data EUID=0 PID=3796753 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/.cipher-key-web-universe-main-dev
[2026-02-05 04:57:46 UTC] USER=www-data EUID=0 PID=3796771 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-02-05 04:57:46 UTC] USER=www-data EUID=0 PID=3796781 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key
[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-02-05 04:57:46 UTC] USER=www-data EUID=0 PID=3796790 ACTION=fsop ARGS=find /data/postgresql/17/web-universe-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-02-05 04:57:46 UTC] USER=www-data EUID=0 PID=3796800 ACTION=fsop ARGS=chown -R postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed
[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-02-05 04:57:46 UTC] USER=www-data EUID=0 PID=3796809 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-02-05 04:57:46 UTC] USER=www-data EUID=0 PID=3796818 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[INFO] ✅ Spool directory created
[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-02-05 04:57:46 UTC] USER=www-data EUID=0 PID=3796836 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/coordinator/PG_VERSION
[2026-02-05 04:57:46 UTC] USER=www-data EUID=0 PID=3796846 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running
[INFO] 5️⃣ Initializing pgBackRest stanza...
[2026-02-05 04:57:46 UTC] USER=www-data EUID=0 PID=3796885 ACTION=fsop ARGS=rm -rf /var/lib/pgbackrest/backup/web-universe-main-dev/backup/web-universe-main-dev-coordinator
[INFO] Creating stanza: web-universe-main-dev-coordinator...
2026-02-05 04:57:46.824 P00 INFO: stanza-create command begin 2.56.0: --exec-id=3796910-19dc6f15 --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 04:57:46.846 P00 INFO: stanza-create for stanza 'web-universe-main-dev-coordinator' on repo1
2026-02-05 04:57:46.858 P00 INFO: stanza-create command end: completed successfully (41ms)
[INFO] ✅ Stanza created successfully
[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
pg_reload_conf
----------------
t
(1 row)
[INFO] ✅ WAL archiving configured for coordinator
[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-02-05 04:57:47 UTC] USER=www-data EUID=0 PID=3796939 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-02-05 04:57:49 UTC] USER=www-data EUID=0 PID=3797047 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 04:57:53 UTC] USER=www-data EUID=0 PID=3797146 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled
[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-02-05 04:57:53 UTC] USER=www-data EUID=0 PID=3797170 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator --log-level-console=info check
2026-02-05 04:57:53.592 P00 INFO: check command begin 2.56.0: --exec-id=3797178-a781b53b --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 04:57:53.612 P00 INFO: check repo1 configuration (primary)
2026-02-05 04:57:53.656 P00 INFO: check repo1 archive for WAL (primary)
2026-02-05 04:57:53.957 P00 INFO: WAL segment 000000010000000000000002 successfully archived to '/var/lib/pgbackrest/backup/web-universe-main-dev/archive/web-universe-main-dev-coordinator/17-1/0000000100000000/000000010000000000000002-1acdf8000e393301527a87cfe74ba7af10f4d2e5.lz4' on repo1
2026-02-05 04:57:53.957 P00 INFO: check command end: completed successfully (368ms)
[INFO] ✅ Stanza verification passed
[INFO] 8️⃣ Creating backup automation scripts...
[2026-02-05 04:57:54 UTC] USER=www-data EUID=0 PID=3797220 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 04:57:54 UTC] USER=www-data EUID=0 PID=3797229 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 04:57:54 UTC] USER=www-data EUID=0 PID=3797251 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[2026-02-05 04:57:54 UTC] USER=www-data EUID=0 PID=3797260 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[INFO] ✅ Backup scripts created
[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-02-05 04:57:54 UTC] USER=www-data EUID=0 PID=3797287 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-web-universe-main-dev
[INFO] ✅ Cron jobs configured
[INFO] Schedule:
[INFO] - Full backup: Sundays at 2:00 AM
[INFO] - Differential backup: Mon-Sat at 2:00 AM
[INFO] 🔟 Creating restore documentation...
[2026-02-05 04:57:54 UTC] USER=www-data EUID=0 PID=3797307 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 04:57:54 UTC] USER=www-data EUID=0 PID=3797321 ACTION=fsop ARGS=sed -i s|__ENV_ID__|web-universe-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 04:57:54 UTC] USER=www-data EUID=0 PID=3797330 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/data/postgresql/17/web-universe-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 04:57:54 UTC] USER=www-data EUID=0 PID=3797339 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 04:57:54 UTC] USER=www-data EUID=0 PID=3797351 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-02-05 04:57:54.558 P00 INFO: start command begin 2.56.0: --exec-id=3797375-96ca0efc --log-level-console=info --log-level-file=debug --stanza=web-universe-main-dev-coordinator
2026-02-05 04:57:54.558 P00 WARN: stop file does not exist for stanza web-universe-main-dev-coordinator
2026-02-05 04:57:54.558 P00 INFO: start command end: completed successfully (4ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-02-05 04:57:54.636 P00 INFO: stanza-upgrade command begin 2.56.0: --exec-id=3797402-aa50b619 --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 04:57:54.637 P00 INFO: stanza-upgrade for stanza 'web-universe-main-dev-coordinator' on repo1
2026-02-05 04:57:54.638 P00 INFO: stanza 'web-universe-main-dev-coordinator' on repo1 is already up to date
2026-02-05 04:57:54.638 P00 INFO: stanza-upgrade command end: completed successfully (8ms)
[INFO] This may take a few minutes depending on database size...
[2026-02-05 04:57:54 UTC] USER=www-data EUID=0 PID=3797407 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260205-045754.log
[2026-02-05 04:57:54 UTC] USER=www-data EUID=0 PID=3797416 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260205-045754.log
[2026-02-05 04:57:54 UTC] USER=www-data EUID=0 PID=3797427 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260205-045754.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-02-05 04:58:04 UTC] USER=www-data EUID=0 PID=3797939 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-3796337.log /var/log/pgbackrest/initial-backup-20260205-045754.log
[INFO] ✅ Initial full backup completed successfully
[INFO] Log: /var/log/pgbackrest/initial-backup-20260205-045754.log
2026-02-05 04:58:04.802 P00 INFO: new backup label = 20260205-045754F
2026-02-05 04:58:04.917 P00 INFO: full backup size = 37.7MB, file total = 1936
2026-02-05 04:58:04.917 P00 INFO: backup command end: completed successfully (10149ms)
2026-02-05 04:58:04.917 P00 INFO: expire command begin 2.56.0: --exec-id=3797444-9a25db73 --log-level-console=info --log-level-file=debug --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --repo1-retention-diff=4 --repo1-retention-full=4 --stanza=web-universe-main-dev-coordinator
2026-02-05 04:58:04.918 P00 INFO: expire command end: completed successfully (1ms)
[INFO] Current backups:
stanza: web-universe-main-dev-coordinator
status: ok
cipher: aes-256-cbc
db (current)
wal archive min/max (17): 000000010000000000000002/000000010000000000000003
full backup: 20260205-045754F
timestamp start/stop: 2026-02-05 04:57:54+00 / 2026-02-05 04:58:04+00
wal start/stop: 000000010000000000000004 / 000000010000000000000004
database size: 37.7MB, database backup size: 37.7MB
repo1: backup set size: 5.7MB, backup size: 5.7MB
[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️ No worker identifier provided - skipping worker backup setup
[INFO] (Run with 'worker-01', 'worker-02', etc. to configure worker backups)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Completed steps:
[INFO] 1. pgBackRest installed and configured
[INFO] 2. WAL archiving enabled (archive_mode=on)
[INFO] 3. PostgreSQL restarted with new settings
[INFO] 4. pgBackRest stanza initialized and verified
[INFO] 5. Initial full backup completed
[INFO] 6. Automated backup cron jobs configured
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO] Coordinator:
[INFO] Stanza: web-universe-main-dev-coordinator
[INFO] Schedule: Full: Sun 2AM, Diff: Mon-Sat 2AM
[INFO] Common:
[INFO] Backup dir: /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO] Archive dir: /var/lib/pgbackrest/archive/web-universe-main-dev
[INFO] Config: /etc/pgbackrest/pgbackrest.conf
[INFO] Restore guide: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] Retention:
[INFO] Full backups: 4 (keep last 4 full backups)
[INFO] Differential: 4 (keep last 4 diff per full)
[INFO] Archive WAL: Auto-managed by pgBackRest
[INFO] Manual commands:
[INFO] Coordinator: sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator backup
[INFO] List all backups: sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up worker backups for 1 worker(s)...
[0;34m[INFO][0m Setting up backup for: worker-01
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for web-universe-main-dev...
[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO] Version: pgBackRest 2.56.0
[INFO] 2️⃣ Creating backup directories...
[2026-02-05 04:58:05 UTC] USER=www-data EUID=0 PID=3798016 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 04:58:05 UTC] USER=www-data EUID=0 PID=3798028 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/web-universe-main-dev
[2026-02-05 04:58:06 UTC] USER=www-data EUID=0 PID=3798041 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-02-05 04:58:06 UTC] USER=www-data EUID=0 PID=3798052 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-02-05 04:58:06 UTC] USER=www-data EUID=0 PID=3798062 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-02-05 04:58:06 UTC] USER=www-data EUID=0 PID=3798071 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-02-05 04:58:08 UTC] USER=www-data EUID=0 PID=3798220 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-02-05 04:58:08 UTC] USER=www-data EUID=0 PID=3798229 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-02-05 04:58:08 UTC] USER=www-data EUID=0 PID=3798238 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-02-05 04:58:08 UTC] USER=www-data EUID=0 PID=3798247 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 04:58:08 UTC] USER=www-data EUID=0 PID=3798256 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO] ✅ Backup directories created
[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-web-universe-main-dev
[2026-02-05 04:58:08 UTC] USER=www-data EUID=0 PID=3798281 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-02-05 04:58:09 UTC] USER=www-data EUID=0 PID=3798291 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key
[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-02-05 04:58:09 UTC] USER=www-data EUID=0 PID=3798300 ACTION=fsop ARGS=find /data/postgresql/17/web-universe-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-02-05 04:58:09 UTC] USER=www-data EUID=0 PID=3798325 ACTION=fsop ARGS=chown -R postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed
[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-02-05 04:58:09 UTC] USER=www-data EUID=0 PID=3798347 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-02-05 04:58:09 UTC] USER=www-data EUID=0 PID=3798356 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created
[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-02-05 04:58:09 UTC] USER=www-data EUID=0 PID=3798365 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/coordinator/PG_VERSION
[2026-02-05 04:58:09 UTC] USER=www-data EUID=0 PID=3798383 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running
[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza web-universe-main-dev-coordinator already initialized and verified
[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
pg_reload_conf
----------------
t
(1 row)
[INFO] ✅ WAL archiving configured for coordinator
[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-02-05 04:58:10 UTC] USER=www-data EUID=0 PID=3798462 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-02-05 04:58:12 UTC] USER=www-data EUID=0 PID=3798512 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 04:58:16 UTC] USER=www-data EUID=0 PID=3798659 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled
[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-02-05 04:58:16 UTC] USER=www-data EUID=0 PID=3798698 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator --log-level-console=info check
2026-02-05 04:58:16.895 P00 INFO: check command begin 2.56.0: --exec-id=3798705-417b46ff --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 04:58:16.914 P00 INFO: check repo1 configuration (primary)
2026-02-05 04:58:16.984 P00 INFO: check repo1 archive for WAL (primary)
2026-02-05 04:58:17.285 P00 INFO: WAL segment 000000010000000000000006 successfully archived to '/var/lib/pgbackrest/backup/web-universe-main-dev/archive/web-universe-main-dev-coordinator/17-1/0000000100000000/000000010000000000000006-afc86b979017f1597729e8855d18561e0f8e87c5.lz4' on repo1
2026-02-05 04:58:17.285 P00 INFO: check command end: completed successfully (401ms)
[INFO] ✅ Stanza verification passed
[INFO] 8️⃣ Creating backup automation scripts...
[2026-02-05 04:58:17 UTC] USER=www-data EUID=0 PID=3798732 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 04:58:17 UTC] USER=www-data EUID=0 PID=3798741 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 04:58:17 UTC] USER=www-data EUID=0 PID=3798759 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[2026-02-05 04:58:17 UTC] USER=www-data EUID=0 PID=3798768 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[INFO] ✅ Backup scripts created
[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-02-05 04:58:17 UTC] USER=www-data EUID=0 PID=3798786 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-web-universe-main-dev
[INFO] ✅ Cron jobs configured
[INFO] Schedule:
[INFO] - Full backup: Sundays at 2:00 AM
[INFO] - Differential backup: Mon-Sat at 2:00 AM
[INFO] 🔟 Creating restore documentation...
[2026-02-05 04:58:17 UTC] USER=www-data EUID=0 PID=3798812 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 04:58:17 UTC] USER=www-data EUID=0 PID=3798827 ACTION=fsop ARGS=sed -i s|__ENV_ID__|web-universe-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 04:58:17 UTC] USER=www-data EUID=0 PID=3798847 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/data/postgresql/17/web-universe-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 04:58:17 UTC] USER=www-data EUID=0 PID=3798875 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-02-05 04:58:17.862 P00 INFO: start command begin 2.56.0: --exec-id=3798903-33f97b18 --log-level-console=info --log-level-file=debug --stanza=web-universe-main-dev-coordinator
2026-02-05 04:58:17.863 P00 WARN: stop file does not exist for stanza web-universe-main-dev-coordinator
2026-02-05 04:58:17.863 P00 INFO: start command end: completed successfully (5ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-02-05 04:58:17.954 P00 INFO: stanza-upgrade command begin 2.56.0: --exec-id=3798914-2a4b6937 --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 04:58:17.955 P00 INFO: stanza-upgrade for stanza 'web-universe-main-dev-coordinator' on repo1
2026-02-05 04:58:17.956 P00 INFO: stanza 'web-universe-main-dev-coordinator' on repo1 is already up to date
2026-02-05 04:58:17.956 P00 INFO: stanza-upgrade command end: completed successfully (8ms)
[INFO] This may take a few minutes depending on database size...
[2026-02-05 04:58:17 UTC] USER=www-data EUID=0 PID=3798918 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260205-045817.log
[2026-02-05 04:58:18 UTC] USER=www-data EUID=0 PID=3798929 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260205-045817.log
[2026-02-05 04:58:18 UTC] USER=www-data EUID=0 PID=3798938 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260205-045817.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-02-05 04:58:24 UTC] USER=www-data EUID=0 PID=3799130 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-3797975.log /var/log/pgbackrest/initial-backup-20260205-045817.log
[INFO] ✅ Initial full backup completed successfully
[INFO] Log: /var/log/pgbackrest/initial-backup-20260205-045817.log
2026-02-05 04:58:24.711 P00 INFO: new backup label = 20260205-045818F
2026-02-05 04:58:24.788 P00 INFO: full backup size = 37.7MB, file total = 1936
2026-02-05 04:58:24.788 P00 INFO: backup command end: completed successfully (6600ms)
2026-02-05 04:58:24.788 P00 INFO: expire command begin 2.56.0: --exec-id=3798955-9ba76fb9 --log-level-console=info --log-level-file=debug --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --repo1-retention-diff=4 --repo1-retention-full=4 --stanza=web-universe-main-dev-coordinator
2026-02-05 04:58:24.788 P00 INFO: expire command end: completed successfully (0ms)
[INFO] Current backups:
stanza: web-universe-main-dev-coordinator
status: ok
cipher: aes-256-cbc
db (current)
wal archive min/max (17): 000000010000000000000002/000000010000000000000006
full backup: 20260205-045754F
timestamp start/stop: 2026-02-05 04:57:54+00 / 2026-02-05 04:58:04+00
wal start/stop: 000000010000000000000004 / 000000010000000000000004
database size: 37.7MB, database backup size: 37.7MB
repo1: backup set size: 5.7MB, backup size: 5.7MB
full backup: 20260205-045818F
timestamp start/stop: 2026-02-05 04:58:18+00 / 2026-02-05 04:58:24+00
wal start/stop: 000000010000000000000007 / 000000010000000000000007
database size: 37.7MB, database backup size: 37.7MB
repo1: backup set size: 5.7MB, backup size: 5.7MB
[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️ No worker identifier provided - skipping worker backup setup
[INFO] (Run with 'worker-01', 'worker-02', etc. to configure worker backups)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Completed steps:
[INFO] 1. pgBackRest installed and configured
[INFO] 2. WAL archiving enabled (archive_mode=on)
[INFO] 3. PostgreSQL restarted with new settings
[INFO] 4. pgBackRest stanza initialized and verified
[INFO] 5. Initial full backup completed
[INFO] 6. Automated backup cron jobs configured
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO] Coordinator:
[INFO] Stanza: web-universe-main-dev-coordinator
[INFO] Schedule: Full: Sun 2AM, Diff: Mon-Sat 2AM
[INFO] Common:
[INFO] Backup dir: /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO] Archive dir: /var/lib/pgbackrest/archive/web-universe-main-dev
[INFO] Config: /etc/pgbackrest/pgbackrest.conf
[INFO] Restore guide: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] Retention:
[INFO] Full backups: 4 (keep last 4 full backups)
[INFO] Differential: 4 (keep last 4 diff per full)
[INFO] Archive WAL: Auto-managed by pgBackRest
[INFO] Manual commands:
[INFO] Coordinator: sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator backup
[INFO] List all backups: sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✓[0m ✅ Backup setup completed for coordinator and all workers
[0;34m[INFO][0m Skipping 06-distribute-tables-canary.sh (test script - set RUN_TESTS=true to enable)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 07-distribute-tables.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 04:58:26 UTC] USER=unknown EUID=33 PID=3799237 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 04:58:26 UTC] USER=unknown EUID=33 PID=3799244 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 04:58:26 UTC] USER=unknown EUID=33 PID=3799251 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 04:58:26 UTC] USER=unknown EUID=33 PID=3799258 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS TABLE DISTRIBUTION
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m 🔐 Secure connection established
[0;34m[INFO][0m Host: db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432
[0;34m[INFO][0m Database: fastorder_web_universe_main_dev_db
[0;34m[INFO][0m SSL: verify-full (TLS 1.2+)
[0;34m[INFO][0m Timeouts: statement=120s, idle_tx=300s
[0;34m[INFO][0m 🔍 Running preflight checks...
[0;34m[INFO][0m Testing database connectivity...
[0;32m[OK][0m ✅ Database connection successful
[0;32m[OK][0m ✅ Connected to correct database: fastorder_web_universe_main_dev_db
[0;34m[INFO][0m Checking Citus extension in database fastorder_web_universe_main_dev_db...
[0;32m[OK][0m Citus version: 13.2-1
[0;34m[INFO][0m Checking worker registration...
[0;32m[OK][0m Registered workers: 1
[0;34m[INFO][0m Worker nodes:
[0;34m[INFO][0m nodename | nodeport | isactive | noderole
[0;34m[INFO][0m -------------------------------------------------------------+----------+----------+----------
[0;34m[INFO][0m db-web-universe-main-dev-postgresql-worker-01.fastorder.com | 5432 | t | primary
[0;34m[INFO][0m (1 row)
[0;34m[INFO][0m
[0;34m[INFO][0m 📊 Starting table distribution...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Distributing: auth.login_account
[0;34m[INFO][0m Description: User authentication table - distributed by region for tenant isolation
[0;34m[INFO][0m Shard key: region_hint
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ⏭️ Table does not exist, skipping
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ All tables distributed successfully!
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m 📊 Citus Cluster Summary:
[0;34m[INFO][0m Distributed tables:
[0;34m[INFO][0m table | type | shard_key | shards | size
[0;34m[INFO][0m ------------------------+-------------+-----------+--------+---------
[0;34m[INFO][0m web.contract_key | reference | <none> | 1 | 16 kB
[0;34m[INFO][0m web.contract_type | reference | <none> | 1 | 16 kB
[0;34m[INFO][0m web.contract_term_json | distributed | id | 32 | 512 kB
[0;34m[INFO][0m web.contract_term_vars | distributed | id | 32 | 1792 kB
[0;34m[INFO][0m (4 rows)
[0;34m[INFO][0m
[0;34m[INFO][0m Worker capacity:
[0;34m[INFO][0m worker | total_shards | total_size
[0;34m[INFO][0m --------+--------------+------------
[0;34m[INFO][0m (0 rows)
[0;34m[INFO][0m
[0;32m[OK][0m Citus table distribution complete
[0;34m[INFO][0m Skipping 08-distribute-tables-rollback.sh (rollback script - run manually only)
[0;34m[INFO][0m Skipping 09-distribute-tables-test.sh (test script - set RUN_TESTS=true to enable)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 10-setup-cdc.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m CDC PIPELINE SETUP (Debezium + Elasticsearch Sink)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Log file: /var/log/fastorder/cdc/10-setup-cdc-*.log
[0;34m[INFO][0m Running CDC setup for identifier: coordinator
[2026-02-05 04:58:31] ==========================================
[2026-02-05 04:58:31] CDC SETUP SCRIPT STARTED
[2026-02-05 04:58:31] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260205_045831.log
[2026-02-05 04:58:31] ==========================================
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 04:58:31] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 04:58:31] CDC Pipeline Setup (Debezium + ES Sink)
[2026-02-05 04:58:31] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 04:58:31] Environment: web-universe-main-dev
[2026-02-05 04:58:31] Identifier: coordinator
[2026-02-05 04:58:31] Service: web
[2026-02-05 04:58:31] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 04:58:31] 📂 CDC_BASE_DIR exists: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc
[2026-02-05 04:58:31] Looking for service folder: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web
[2026-02-05 04:58:31]
[2026-02-05 04:58:31] 📂 Found CDC configuration for service: web
[2026-02-05 04:58:31] Scanning for subservice directories in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web
[2026-02-05 04:58:31] Found subservice: config, checking for steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps
[2026-02-05 04:58:31]
[2026-02-05 04:58:31] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 04:58:31] Setting up CDC for: web/config
[2026-02-05 04:58:31] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 04:58:31] Found 3 step script(s) in /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps
[2026-02-05 04:58:31]
[2026-02-05 04:58:31] 🔧 Running: 01-setup-config-cdc.sh
[2026-02-05 04:58:31] Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps/01-setup-config-cdc.sh
[2026-02-05 04:58:31] Executing directly (script is executable)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Config CDC Setup - Automatic Role Detection
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service: web
[INFO] Zone: universe
[INFO] Branch: main
[INFO] Environment: dev
[INFO] Identifier: coordinator
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service 'web' requires config.* schema
[INFO] CDC Role for web in zone universe: master
[INFO] ════════════════════════════════════════════════════════════
[INFO] CONTROL PLANE MASTER (zone=universe)
[INFO] Setting up Debezium CDC Publisher
[INFO] ════════════════════════════════════════════════════════════
[INFO] Executing Debezium config setup...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Debezium Config CDC Master Setup
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: web-universe-main-dev
[INFO] PostgreSQL Host: db-web-universe-main-dev-postgresql.fastorder.com
[INFO] Database: fastorder_web_universe_main_dev_db
[INFO] Connector: debezium-web-universe-main-dev-config
[INFO] Topic Prefix: cdc.web_universe_main_dev
[INFO] Replication Slot: dbz_web_universe_main_dev_config
[INFO] Tables: config.public_defaults,config.feature_flags,config.config_version
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Step 1: Creating PostgreSQL publication for config tables...
[ OK ] Debezium config CDC master setup complete
[INFO] No topology.json found at /opt/fastorder/state/web-universe-main-dev/topology.json - skipping merge
[INFO]
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Config CDC Setup Complete
[INFO] ═══════════════════════════════════════════════════════════════
═══════════════════════════════════════════════════════════════
Service Capabilities: web-universe-main-dev
═══════════════════════════════════════════════════════════════
Service: web
Zone: universe
Branch: main
Environment: dev
Config Schema: ✅ YES
Redis Cache: ✅ YES
CDC Role: master
CDC Master Configuration:
Debezium: debezium-web-universe-main-dev-config
Topic Prefix: cdc.web_universe_main_dev
Repl Slot: dbz_web_universe_main_dev_config
Tables: config.public_defaults,config.feature_flags,config.config_version
Required Schemas: config tenant dashboard environment resource service item company communication ai
═══════════════════════════════════════════════════════════════
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]
[INFO] Log file: /var/log/fastorder/cdc/config-cdc-20260205_045831.log
[ OK ] Config CDC setup finished successfully
[2026-02-05 04:58:33] ✅ Completed: 01-setup-config-cdc.sh
[2026-02-05 04:58:33]
[2026-02-05 04:58:33] 🔧 Running: 02-setup-debezium-config.sh
[2026-02-05 04:58:33] Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps/02-setup-debezium-config.sh
[2026-02-05 04:58:33] Executing directly (script is executable)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Debezium Config CDC Master Setup
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: web-universe-main-dev
[INFO] PostgreSQL Host: db-web-universe-main-dev-postgresql.fastorder.com
[INFO] Database: fastorder_web_universe_main_dev_db
[INFO] Connector: debezium-web-universe-main-dev-config
[INFO] Topic Prefix: cdc.web_universe_main_dev
[INFO] Replication Slot: dbz_web_universe_main_dev_config
[INFO] Tables: config.public_defaults,config.feature_flags,config.config_version
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Step 1: Creating PostgreSQL publication for config tables...
[2026-02-05 04:58:33] ❌ FAILED: 02-setup-debezium-config.sh (exit code: 2)
[2026-02-05 04:58:33] ❌ CRITICAL: This is a required step for CDC pipeline. Aborting.
[0;31m[ERROR][0m ❌ Database infrastructure (postgresql) setup failed with exit code: 2
⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...