Environment: Web Universe Main Dev on web-03
"{\"env\": \"dev\", \"zone\": \"universe\", \"branch\": \"main\", \"db_app\": \"postgresql\", \"service\": \"web\", \"db_enabled\": true, \"pg_standby\": 0, \"pg_workers\": 1, \"description\": \"\", \"iam_enabled\": false, \"worker_1_ip\": \"\", \"eventbus_app\": \"kafka\", \"worker_1_fqdn\": \"db-web-universe-main-dev-postgresql-worker-01.fastorder.com\", \"search_enabled\": false, \"eventbus_enabled\": true, \"postgresql_enabled\": true, \"postgresql_run_verification\": true}"
This job encountered an error. You can restart from the failed step.
This job has been restarted. You are viewing an older attempt. The logs and status shown below are from the latest retry.
This job failed at one of the steps below. You can resume from where it failed to save time and avoid re-running successful steps.
⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...[0;34m[INFO][0m Using eventbus engine from EVENTBUS_ENGINE environment variable: kafka
[0;34m[INFO][0m Cleaning up any existing locks...
[0;32m[1mStarting eventbus engine: kafka[0m
[1;33m═══════════════════════════════════════════════[0m
[0;34m[INFO][0m Using environment from web interface: web-universe-main-dev
[0;32m[2026-02-05 06:28:13][0m Using web-provided environment: web-universe-main-dev
[0;32m[2026-02-05 06:28:13][0m Service: web, Zone: universe, Branch: main, Env: dev
[0;32m✓[0m Environment initialized successfully (mode: general)
[0;34m[INFO][0m Starting Kafka setup process...
[0;34m[INFO][0m Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Found 13 step(s) to execute
[0;34m[INFO][0m 📦 Step 1/13: install debezium connector...
═══════════════════════════════════════════════════════════════════
Fetching latest versions from Maven Central...
Installing Debezium PostgreSQL Connector
Debezium version: 3.4.1.Final
pgjdbc version: 42.7.9
═══════════════════════════════════════════════════════════════════
[OK] Debezium 3.4.1.Final with pgjdbc 42.7.9 already installed
[0;32m[OK][0m ✅ Step 1 completed: 00-install-debezium-connector.sh
[0;34m[INFO][0m 📦 Step 2/13: kafka setup...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials for secrets vault...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
🧹 Checking for orphaned Kafka processes on ports 9092, 9093, 8083...
⚠️ Found process on port 9092 (PIDs: [2026-02-05 06:28:14 UTC] USER=www-data EUID=0 PID=4084432 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 2>/dev/null || true
3749380
3750794
3750795
3750796
3750797
3750798
3754712
3761495), killing...
[2026-02-05 06:28:15 UTC] USER=www-data EUID=0 PID=4084453 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 06:28:14 UTC] USER=www-data EUID=0 PID=4084432 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 2>/dev/null || true
3749380
3750794
3750795
3750796
3750797
3750798
3754712
3761495 2>/dev/null || true
/usr/bin/bash: line 2: 3749380: command not found
/usr/bin/bash: line 3: 3750794: command not found
/usr/bin/bash: line 4: 3750795: command not found
/usr/bin/bash: line 5: 3750796: command not found
/usr/bin/bash: line 6: 3750797: command not found
/usr/bin/bash: line 7: 3750798: command not found
/usr/bin/bash: line 8: 3754712: command not found
⚠️ Found process on port 9093 (PIDs: [2026-02-05 06:28:16 UTC] USER=www-data EUID=0 PID=4084556 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 2>/dev/null || true
3750492
3750794
3750796
3761495), killing...
[2026-02-05 06:28:16 UTC] USER=www-data EUID=0 PID=4084587 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 06:28:16 UTC] USER=www-data EUID=0 PID=4084556 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 2>/dev/null || true
3750492
3750794
3750796
3761495 2>/dev/null || true
/usr/bin/bash: line 2: 3750492: command not found
/usr/bin/bash: line 3: 3750794: command not found
/usr/bin/bash: line 4: 3750796: command not found
⚠️ Found process on port 8083 (PIDs: [2026-02-05 06:28:17 UTC] USER=www-data EUID=0 PID=4084620 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 2>/dev/null || true
3749380
3750795
3750797), killing...
[2026-02-05 06:28:17 UTC] USER=www-data EUID=0 PID=4084636 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 06:28:17 UTC] USER=www-data EUID=0 PID=4084620 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 2>/dev/null || true
3749380
3750795
3750797 2>/dev/null || true
/usr/bin/bash: line 2: 3749380: command not found
/usr/bin/bash: line 3: 3750795: command not found
✅ Port cleanup completed
Ensuring KAFKA application environment for coordinator...
[0;34m[INFO][0m Using existing KAFKA environment: eventbus-web-universe-main-dev-kafka-connect.fastorder.com (10.100.1.75)
Ensuring KAFKA_BROKER_IP application environment for coordinator...
[0;34m[INFO][0m Using existing KAFKA BROKER environment: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com (10.100.1.225)
[0;34m[INFO][0m Kafka Broker IP: 10.100.1.225
[0;34m[INFO][0m Kafka Connect IP: 10.100.1.75
[0;34m[INFO][0m Registered /etc/hosts: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com -> 10.100.1.225
[0;34m[INFO][0m Registered /etc/hosts: eventbus-web-universe-main-dev-kafka-connect.fastorder.com -> 10.100.1.75
🔐 Initializing keystore passwords...
[0;34m[INFO][0m 🔍 Checking secrets backend (provider: aws)...
[0;32m✅ Retrieved passwords from remote backend[0m
[0;34m[INFO][0m ✅ Using existing passwords from backend
✅ Keystore passwords initialized
- Keystore password: HGvJkWmj... (32 chars)
- Truststore password: sZRdI2nT... (32 chars)
[0;34m[INFO][0m 🔐 Vaulting kafka passwords to remote backend...
[0;32m✅ Passwords vaulted to remote backend[0m
✅ Kafka keystore passwords saved to AWS Secrets Manager
[INFO] Generating for: web-universe-main-dev (host=eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com ip=10.100.1.225)
[2026-02-05 06:28:26 UTC] USER=www-data EUID=0 PID=4085599 ACTION=fsop ARGS=rm -rf /opt/kafka/secrets/web-universe-main-dev/coordinator /data/kafka/web-universe-main-dev/coordinator
[2026-02-05 06:28:26 UTC] USER=www-data EUID=0 PID=4085619 ACTION=fsop ARGS=mkdir -p /opt/kafka/secrets/web-universe-main-dev/coordinator /opt/kafka/config/web-universe-main-dev/coordinator /opt/kafka/secrets/web-universe-main-dev/coordinator/pem /data/kafka/web-universe-main-dev_coordinator-data
[2026-02-05 06:28:26 UTC] USER=www-data EUID=0 PID=4085635 ACTION=fsop ARGS=chown -R kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator
[2026-02-05 06:28:26 UTC] USER=www-data EUID=0 PID=4085670 ACTION=fsop ARGS=chown -R kafka:kafka /opt/kafka/config/web-universe-main-dev/coordinator /data/kafka/web-universe-main-dev_coordinator-data
[2026-02-05 06:28:26 UTC] USER=www-data EUID=0 PID=4085702 ACTION=fsop ARGS=chmod 770 /opt/kafka/config/web-universe-main-dev/coordinator /data/kafka/web-universe-main-dev_coordinator-data
[2026-02-05 06:28:27 UTC] USER=www-data EUID=0 PID=4085719 ACTION=fsop ARGS=chmod 750 /opt/kafka/secrets/web-universe-main-dev/coordinator
[2026-02-05 06:28:27 UTC] USER=www-data EUID=0 PID=4085753 ACTION=fsop ARGS=chmod 750 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
[2026-02-05 06:28:27 UTC] USER=www-data EUID=0 PID=4085781 ACTION=fsop ARGS=chmod 700 /tmp/fo-tls.8kogCq
[2026-02-05 06:28:27 UTC] USER=www-data EUID=0 PID=4085803 ACTION=fsop ARGS=chmod 755 /tmp/fo-tls.8kogCq
[2026-02-05 06:28:27 UTC] USER=www-data EUID=0 PID=4085816 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/fo-tls.8kogCq/ra_root.crt
[2026-02-05 06:28:27 UTC] USER=www-data EUID=0 PID=4085835 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/fo-tls.8kogCq/ra_root.key
[2026-02-05 06:28:27 UTC] USER=www-data EUID=0 PID=4085854 ACTION=fsop ARGS=chmod 644 /tmp/fo-tls.8kogCq/ra_root.crt
[2026-02-05 06:28:27 UTC] USER=www-data EUID=0 PID=4085874 ACTION=fsop ARGS=chmod 644 /tmp/fo-tls.8kogCq/ra_root.key
Certificate was added to keystore
[2026-02-05 06:28:28 UTC] USER=www-data EUID=0 PID=4086113 ACTION=fsop ARGS=mv /tmp/fo-tls.8kogCq/truststore.jks /opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks
[2026-02-05 06:28:28 UTC] USER=www-data EUID=0 PID=4086132 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks
[2026-02-05 06:28:28 UTC] USER=www-data EUID=0 PID=4086142 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks
Generating 4,096 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 825 days
for: CN=eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com, OU=Kafka Broker, O=FastOrder, C=AE
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /tmp/fo-tls.8kogCq/kafka.server.keystore.jks -destkeystore /tmp/fo-tls.8kogCq/kafka.server.keystore.jks -deststoretype pkcs12".
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /tmp/fo-tls.8kogCq/kafka.server.keystore.jks -destkeystore /tmp/fo-tls.8kogCq/kafka.server.keystore.jks -deststoretype pkcs12".
Certificate request self-signature ok
subject=C = AE, O = FastOrder, OU = Kafka Broker, CN = eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com
Certificate was added to keystore
Warning:
Certificate reply was installed in keystore
Warning:
[2026-02-05 06:28:33 UTC] USER=www-data EUID=0 PID=4086571 ACTION=fsop ARGS=mv /tmp/fo-tls.8kogCq/kafka.server.keystore.jks /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 06:28:33 UTC] USER=www-data EUID=0 PID=4086583 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 06:28:33 UTC] USER=www-data EUID=0 PID=4086596 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.server.keystore.jks
Generating 4,096 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 825 days
for: CN=eventbus-web-universe-main-dev-kafka-connect.fastorder.com, OU=Kafka Connect REST, O=FastOrder, C=AE
Certificate request self-signature ok
subject=C = AE, O = FastOrder, OU = Kafka Connect REST, CN = eventbus-web-universe-main-dev-kafka-connect.fastorder.com
Certificate was added to keystore
Certificate reply was installed in keystore
[2026-02-05 06:28:44 UTC] USER=www-data EUID=0 PID=4087256 ACTION=fsop ARGS=mv /tmp/fo-tls.8kogCq/connect-rest.keystore.p12 /opt/kafka/secrets/web-universe-main-dev/coordinator/connect-rest.keystore.p12
Certificate request self-signature ok
subject=CN = kafka-client-web-universe-main-dev, OU = Kafka Client, O = FastOrder, C = AE
[2026-02-05 06:28:44 UTC] USER=www-data EUID=0 PID=4087297 ACTION=fsop ARGS=cp /tmp/fo-tls.8kogCq/ra_root.crt /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem
[2026-02-05 06:28:44 UTC] USER=www-data EUID=0 PID=4087306 ACTION=fsop ARGS=cp /tmp/fo-tls.8kogCq/client-key.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:28:44 UTC] USER=www-data EUID=0 PID=4087315 ACTION=fsop ARGS=cp /tmp/fo-tls.8kogCq/client-cert.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
[2026-02-05 06:28:44 UTC] USER=www-data EUID=0 PID=4087324 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
[2026-02-05 06:28:44 UTC] USER=www-data EUID=0 PID=4087333 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
[2026-02-05 06:28:45 UTC] USER=www-data EUID=0 PID=4087343 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:28:45 UTC] USER=www-data EUID=0 PID=4087362 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.client.keystore.p12
[2026-02-05 06:28:45 UTC] USER=www-data EUID=0 PID=4087371 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.client.keystore.p12
🔐 Ensuring kafka user has access to PostgreSQL certificates...
✅ kafka is already in postgres group
🧹 Cleaning up conflicting services and processes on Kafka ports on 10.100.1.225...
🔪 Killing processes on 10.100.1.225:8083: [2026-02-05 06:28:45 UTC] USER=www-data EUID=0 PID=4087410 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:8083 -t 2>/dev/null || true
[2026-02-05 06:28:45 UTC] USER=www-data EUID=0 PID=4087473 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 06:28:45 UTC] USER=www-data EUID=0 PID=4087410 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:8083 -t 2>/dev/null || true
🔪 Killing processes on 10.100.1.225:9092: [2026-02-05 06:28:45 UTC] USER=www-data EUID=0 PID=4087487 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:9092 -t 2>/dev/null || true
3761495
[2026-02-05 06:28:46 UTC] USER=www-data EUID=0 PID=4087570 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 06:28:45 UTC] USER=www-data EUID=0 PID=4087487 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:9092 -t 2>/dev/null || true
3761495
/usr/bin/bash: line 2: 3761495: command not found
🔪 Killing processes on 10.100.1.225:9093: [2026-02-05 06:28:46 UTC] USER=www-data EUID=0 PID=4087590 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:9093 -t 2>/dev/null || true
3761495
[2026-02-05 06:28:46 UTC] USER=www-data EUID=0 PID=4087615 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 06:28:46 UTC] USER=www-data EUID=0 PID=4087590 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:9093 -t 2>/dev/null || true
3761495
/usr/bin/bash: line 2: 3761495: command not found
✅ Port cleanup completed
🔧 Checking for Kafka Connect internal topics with incorrect cleanup policy...
📋 Kafka broker is running, checking topic cleanup policies...
✅ Topic cleanup policy fix completed
🔧 Creating environment-specific systemd units...
🔧 Writing client properties to /etc/kafka/client-web-universe-main-dev-coordinator.properties ...
[2026-02-05 06:28:57 UTC] USER=www-data EUID=0 PID=4089624 ACTION=fsop ARGS=chown root:kafka /etc/kafka/client-web-universe-main-dev-coordinator.properties
[2026-02-05 06:28:57 UTC] USER=www-data EUID=0 PID=4089635 ACTION=fsop ARGS=chmod 0644 /etc/kafka/client-web-universe-main-dev-coordinator.properties
🔧 Creating PEM certificates for PHP mTLS access...
[2026-02-05 06:28:57 UTC] USER=www-data EUID=0 PID=4089644 ACTION=passthru ARGS=bash -c openssl pkcs12 -in '/opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.client.keystore.p12' -clcerts -nokeys -passin pass:'HGvJkWmjjIaZzWVQzIjopYiQoGhZCsRH' -out '/opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.crt' 2>/dev/null
[2026-02-05 06:28:57 UTC] USER=www-data EUID=0 PID=4089654 ACTION=passthru ARGS=bash -c openssl pkcs12 -in '/opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.client.keystore.p12' -nocerts -nodes -passin pass:'HGvJkWmjjIaZzWVQzIjopYiQoGhZCsRH' -out '/opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.key' 2>/dev/null
[2026-02-05 06:28:57 UTC] USER=www-data EUID=0 PID=4089664 ACTION=passthru ARGS=bash -c keytool -exportcert -alias fastorder-ra-root -keystore '/opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks' -storepass 'sZRdI2nTnX4yhzopYtu0ttl9GtAWGH7c' -rfc -file '/opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.crt' 2>/dev/null
[2026-02-05 06:28:57 UTC] USER=www-data EUID=0 PID=4089709 ACTION=fsop ARGS=chown root:kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.crt /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.key /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.crt
[2026-02-05 06:28:57 UTC] USER=www-data EUID=0 PID=4089718 ACTION=fsop ARGS=chmod 0644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.crt /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.crt
[2026-02-05 06:28:57 UTC] USER=www-data EUID=0 PID=4089727 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.key
🔐 Creating connector secrets file for FileConfigProvider...
[2026-02-05 06:28:57 UTC] USER=www-data EUID=0 PID=4089755 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/connector-secrets.properties
[2026-02-05 06:28:57 UTC] USER=www-data EUID=0 PID=4089764 ACTION=fsop ARGS=chmod 0600 /opt/kafka/secrets/web-universe-main-dev/coordinator/connector-secrets.properties
✅ Connector secrets file created: /opt/kafka/secrets/web-universe-main-dev/coordinator/connector-secrets.properties
FileConfigProvider syntax: ${file:/opt/kafka/secrets/web-universe-main-dev/coordinator/connector-secrets.properties:key_name}
🔧 Creating Canary Event timer for pipeline verification...
[2026-02-05 06:28:57 UTC] USER=www-data EUID=0 PID=4089791 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 06:28:58 UTC] USER=www-data EUID=0 PID=4089856 ACTION=passthru ARGS=systemctl enable kafka-canary-web-universe-main-dev.timer
[2026-02-05 06:28:59 UTC] USER=www-data EUID=0 PID=4089918 ACTION=passthru ARGS=systemctl start kafka-canary-web-universe-main-dev.timer
✅ Canary timer installed: kafka-canary-web-universe-main-dev.timer (every 5 minutes)
[2026-02-05 06:28:59 UTC] USER=www-data EUID=0 PID=4089934 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 06:29:05 UTC] USER=www-data EUID=0 PID=4090572 ACTION=passthru ARGS=systemctl mask kafka-server
Failed to print table: Broken pipe
🔒 Adjusting group ownership and permissions ...
[2026-02-05 06:29:08 UTC] USER=www-data EUID=0 PID=4090734 ACTION=fsop ARGS=chown :kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 06:29:08 UTC] USER=www-data EUID=0 PID=4090771 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 06:29:08 UTC] USER=www-data EUID=0 PID=4090847 ACTION=fsop ARGS=chown root:kafka /etc/kafka/client-web-universe-main-dev-coordinator.properties
[2026-02-05 06:29:08 UTC] USER=www-data EUID=0 PID=4090885 ACTION=fsop ARGS=chmod 0644 /etc/kafka/client-web-universe-main-dev-coordinator.properties
✅ Kafka configuration complete for web-universe-main-dev_coordinator
Broker ID : 89
Broker keystore : /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.server.keystore.jks
REST keystore : /opt/kafka/secrets/web-universe-main-dev/coordinator/connect-rest.keystore.p12
Truststore : /opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks
Client PKCS12 : /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.client.keystore.p12
Data directory : /data/kafka/web-universe-main-dev_coordinator-data
Server config : /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
Connect config : /opt/kafka/config/web-universe-main-dev/coordinator/connect-distributed.properties
CLI client config : /etc/kafka/client-web-universe-main-dev-coordinator.properties
🎯 Next step: Run 03-restart-kafka-related-services.sh to start services
[0;32m[OK][0m ✅ Step 2 completed: 01-kafka-setup.sh
[0;34m[INFO][0m 📦 Step 3/13: metadata...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 🎯 Kafka metadata mode: kraft
╔════════════════════════════════════════════════════════════════════╗
║ Kafka Metadata Layer Setup ║
╚════════════════════════════════════════════════════════════════════╝
Environment : web-universe-main-dev
Service : web
Zone : universe
Branch : main
Environment : dev
VM IP : 142.93.238.16
Metadata Mode : kraft
📋 KRaft Mode (Modern)
────────────────────────────────────────────────────────────────
✅ No ZooKeeper dependency
✅ Faster metadata operations
✅ Simplified architecture
✅ Recommended for new deployments
⚠️ Requires Kafka 3.3+ in production
════════════════════════════════════════════════════════════════════
[INFO] 🚀 Executing KRaft setup script...
[INFO] Script: /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps/metadata/kraft.sh
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 06:29:10 UTC] USER=www-data EUID=0 PID=4091216 ACTION=fsop ARGS=chown -R kafka:kafka /data/kafka/web-universe-main-dev_coordinator-meta /opt/kafka/config/web-universe-main-dev/coordinator /data/kafka/web-universe-main-dev_coordinator-data
[2026-02-05 06:29:10 UTC] USER=www-data EUID=0 PID=4091234 ACTION=fsop ARGS=chmod 770 /data/kafka/web-universe-main-dev_coordinator-meta /opt/kafka/config/web-universe-main-dev/coordinator /data/kafka/web-universe-main-dev_coordinator-data
[INFO] Adding eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com to /etc/hosts -> 10.100.1.225
[INFO] Adding eventbus-web-universe-main-dev-kafka-connect.fastorder.com to /etc/hosts -> 10.100.1.75
[INFO] Setting up KRaft for: web-universe-main-dev (host=eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com ip=10.100.1.225)
[2026-02-05 06:29:10 UTC] USER=www-data EUID=0 PID=4091295 ACTION=fsop ARGS=mkdir -p /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev
[INFO] Using existing cluster.id from state
🔧 Configuring Kafka for KRaft mode...
[2026-02-05 06:29:11 UTC] USER=www-data EUID=0 PID=4091334 ACTION=fsop ARGS=test -f /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 06:29:11 UTC] USER=www-data EUID=0 PID=4091352 ACTION=fsop ARGS=test -r /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 06:29:11 UTC] USER=www-data EUID=0 PID=4091362 ACTION=fsop ARGS=sed -i /^zookeeper\.connect=/d /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 06:29:11 UTC] USER=www-data EUID=0 PID=4091379 ACTION=passthru ARGS=bash -c grep -q '^process.roles=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 06:29:11 UTC] USER=www-data EUID=0 PID=4091400 ACTION=passthru ARGS=bash -c grep -q '^node.id=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 06:29:11 UTC] USER=www-data EUID=0 PID=4091478 ACTION=passthru ARGS=bash -c grep -q '^controller.listener.names=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 06:29:11 UTC] USER=www-data EUID=0 PID=4091518 ACTION=passthru ARGS=bash -c grep -q '^controller.quorum.voters=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 06:29:12 UTC] USER=www-data EUID=0 PID=4091555 ACTION=passthru ARGS=bash -c grep -q '^metadata.log.dir=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 06:29:12 UTC] USER=www-data EUID=0 PID=4091606 ACTION=passthru ARGS=bash -c grep -q '^log.dirs=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 06:29:12 UTC] USER=www-data EUID=0 PID=4091618 ACTION=fsop ARGS=sed -i s|^log.dirs=.*|log.dirs=/data/kafka/web-universe-main-dev_coordinator-data| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 06:29:12 UTC] USER=www-data EUID=0 PID=4091640 ACTION=passthru ARGS=bash -c grep -q '^listeners=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 06:29:12 UTC] USER=www-data EUID=0 PID=4091663 ACTION=passthru ARGS=bash -c grep -q 'CONTROLLER://' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 06:29:12 UTC] USER=www-data EUID=0 PID=4091672 ACTION=fsop ARGS=sed -i s|^listeners=.*|listeners=SSL://10.100.1.225:9092,CONTROLLER://10.100.1.225:9093| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 06:29:12 UTC] USER=www-data EUID=0 PID=4091691 ACTION=passthru ARGS=bash -c grep -q '^advertised.listeners=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 06:29:13 UTC] USER=www-data EUID=0 PID=4091762 ACTION=fsop ARGS=sed -i s|^listener.security.protocol.map=.*|listener.security.protocol.map=SSL:SSL,CONTROLLER:PLAINTEXT| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 06:29:13 UTC] USER=www-data EUID=0 PID=4091779 ACTION=passthru ARGS=bash -c grep -q '^inter.broker.listener.name=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 06:29:13 UTC] USER=www-data EUID=0 PID=4091798 ACTION=fsop ARGS=sed -i s|^inter.broker.listener.name=.*|inter.broker.listener.name=SSL| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 06:29:13 UTC] USER=www-data EUID=0 PID=4091816 ACTION=passthru ARGS=bash -c grep -q '^offsets.topic.replication.factor=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 06:29:13 UTC] USER=www-data EUID=0 PID=4091838 ACTION=fsop ARGS=sed -i s|^offsets.topic.replication.factor=.*|offsets.topic.replication.factor=1| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 06:29:13 UTC] USER=www-data EUID=0 PID=4091877 ACTION=fsop ARGS=sed -i s|^transaction.state.log.replication.factor=.*|transaction.state.log.replication.factor=1| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 06:29:13 UTC] USER=www-data EUID=0 PID=4091892 ACTION=passthru ARGS=bash -c grep -q '^transaction.state.log.min.isr=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 06:29:13 UTC] USER=www-data EUID=0 PID=4091915 ACTION=fsop ARGS=sed -i s|^transaction.state.log.min.isr=.*|transaction.state.log.min.isr=1| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
✅ KRaft configuration applied to server.properties
[2026-02-05 06:29:14 UTC] USER=www-data EUID=0 PID=4091946 ACTION=fsop ARGS=test -f /data/kafka/web-universe-main-dev_coordinator-meta/meta.properties
[2026-02-05 06:29:14 UTC] USER=www-data EUID=0 PID=4091967 ACTION=fsop ARGS=test -f /data/kafka/web-universe-main-dev_coordinator-data/meta.properties
[INFO] Already formatted: both /data/kafka/web-universe-main-dev_coordinator-meta and /data/kafka/web-universe-main-dev_coordinator-data have meta.properties
🔧 Creating/refreshing KRaft systemd unit...
[2026-02-05 06:29:14 UTC] USER=www-data EUID=0 PID=4092002 ACTION=passthru ARGS=systemctl daemon-reload
✅ Ensured confluent-kraft-web-universe-main-dev_coordinator.service
🛑 Stopping legacy ZooKeeper-mode services and current KRaft instance...
🛑 Stopping current: confluent-kraft-web-universe-main-dev_coordinator.service
[2026-02-05 06:29:15 UTC] USER=www-data EUID=0 PID=4092125 ACTION=passthru ARGS=systemctl stop confluent-kraft-web-universe-main-dev_coordinator.service
🧹 Cleaning up rogue Kafka processes...
🧹 Killing any processes holding Kafka ports 9092, 9093...
🔪 Killing processes on port 9092: 3750794
3750795
3750796
3750797
3750798
3754712
[2026-02-05 06:29:19 UTC] USER=www-data EUID=0 PID=4092929 ACTION=passthru ARGS=bash -c kill -9 3750794
[2026-02-05 06:29:19 UTC] USER=www-data EUID=0 PID=4092947 ACTION=passthru ARGS=bash -c kill -9 3750795
[2026-02-05 06:29:20 UTC] USER=www-data EUID=0 PID=4092969 ACTION=passthru ARGS=bash -c kill -9 3750796
[2026-02-05 06:29:20 UTC] USER=www-data EUID=0 PID=4092985 ACTION=passthru ARGS=bash -c kill -9 3750797
[2026-02-05 06:29:20 UTC] USER=www-data EUID=0 PID=4093001 ACTION=passthru ARGS=bash -c kill -9 3750798
[2026-02-05 06:29:20 UTC] USER=www-data EUID=0 PID=4093030 ACTION=passthru ARGS=bash -c kill -9 3754712
🔪 Killing processes on port 8083: 3749380
[2026-02-05 06:29:23 UTC] USER=www-data EUID=0 PID=4093438 ACTION=passthru ARGS=bash -c kill -9 3749380
✅ Legacy services stopped and rogue processes cleaned
🔓 Removing stale lock files...
[2026-02-05 06:29:28 UTC] USER=www-data EUID=0 PID=4093996 ACTION=fsop ARGS=test -f /data/kafka/web-universe-main-dev_coordinator-meta/.lock
✅ Lock file check complete
🚀 Starting confluent-kraft-web-universe-main-dev_coordinator.service ...
[2026-02-05 06:29:29 UTC] USER=www-data EUID=0 PID=4094027 ACTION=passthru ARGS=systemctl enable confluent-kraft-web-universe-main-dev_coordinator.service
[2026-02-05 06:29:29 UTC] USER=www-data EUID=0 PID=4094115 ACTION=passthru ARGS=systemctl restart confluent-kraft-web-universe-main-dev_coordinator.service
🔧 Patching shared Connect unit to follow KRaft broker...
[2026-02-05 06:29:32 UTC] USER=www-data EUID=0 PID=4094800 ACTION=fsop ARGS=sed -i -e s|${FULL_ENV}|web-universe-main-dev|g -e s|${IDENTIFIER}|coordinator|g -e s|${CONFIG_DIR}|/opt/kafka/config/web-universe-main-dev/coordinator|g /etc/systemd/system/confluent-connect-web-universe-main-dev_coordinator.service
[2026-02-05 06:29:33 UTC] USER=www-data EUID=0 PID=4094821 ACTION=fsop ARGS=sed -i s|^After=.*|After=network-online.target confluent-kraft-web-universe-main-dev_coordinator.service| /etc/systemd/system/confluent-connect-web-universe-main-dev_coordinator.service
[2026-02-05 06:29:33 UTC] USER=www-data EUID=0 PID=4094836 ACTION=fsop ARGS=sed -i s|^Wants=.*|Wants=confluent-kraft-web-universe-main-dev_coordinator.service| /etc/systemd/system/confluent-connect-web-universe-main-dev_coordinator.service
[2026-02-05 06:29:33 UTC] USER=www-data EUID=0 PID=4094861 ACTION=passthru ARGS=systemctl daemon-reload
✅ Connect unit patched
[2026-02-05 06:29:34 UTC] USER=www-data EUID=0 PID=4094946 ACTION=fsop ARGS=test -f /opt/kafka/config/web-universe-main-dev/coordinator/connect-distributed.properties
[2026-02-05 06:29:34 UTC] USER=www-data EUID=0 PID=4094969 ACTION=fsop ARGS=ln -sf /opt/kafka/config/web-universe-main-dev/coordinator/connect-distributed.properties /opt/kafka/config/connect-distributed.properties
⏳ Waiting for broker coordinator on SSL://eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092 ...
⏳ Waiting for KRaft broker... (attempt 1, 0s/600s)
Debug: Last error was: [2026-02-05 06:29:34 UTC] USER=www-data EUID=0 PID=4095001 ACTION=passthru ARGS=bash -c timeout 5 sudo -u kafka /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server 'eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092' --command-config '/etc/kafka/client-web-universe-main-dev-coordinator.properties' describe --status
⏳ Still waiting... (attempt 10, 61s/600s)
⏳ Still waiting... (attempt 20, 127s/600s)
⏳ Waiting for KRaft broker... (attempt 30, 190s/600s)
Debug: Last error was: [2026-02-05 06:32:44 UTC] USER=www-data EUID=0 PID=4134951 ACTION=passthru ARGS=bash -c timeout 5 sudo -u kafka /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server 'eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092' --command-config '/etc/kafka/client-web-universe-main-dev-coordinator.properties' describe --status
✅ coordinator responded after 222s (attempt 35)
---- server.properties (key lines) ----
[2026-02-05 06:33:49 UTC] USER=www-data EUID=0 PID=4144988 ACTION=passthru ARGS=bash -c grep -E '^(listeners|advertised\.listeners|process\.roles|controller\.quorum\.voters|controller\.listener\.names|inter\.broker\.listener\.name|log\.dirs|metadata\.log\.dir)=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
---------------------------------------
✅ KRaft setup complete for web-universe-main-dev_coordinator
server.properties : /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
data dir : /data/kafka/web-universe-main-dev_coordinator-data
meta dir : /data/kafka/web-universe-main-dev_coordinator-meta
systemd unit : confluent-kraft-web-universe-main-dev_coordinator.service
🔧 Kafka Configuration Modified:
✓ process.roles, node.id, controller.quorum.voters, controller.listener.names
✓ listeners (SSL + CONTROLLER) and advertised.listeners (FQDN fallback to IP)
✓ listener.security.protocol.map, inter.broker.listener.name
✓ log.dirs -> /data/kafka/web-universe-main-dev_coordinator-data, metadata.log.dir -> /data/kafka/web-universe-main-dev_coordinator-meta
✓ removed zookeeper.connect (if present)
✓ created/refreshed dedicated KRaft systemd unit
✓ patched shared Connect unit to follow KRaft broker
✓ symlinked /opt/kafka/config/web-universe-main-dev/coordinator/connect-distributed.properties -> /opt/kafka/config/connect-distributed.properties (compat)
🔎 Check quorum:
/opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092 --command-config /etc/kafka/client-web-universe-main-dev-coordinator.properties describe --status
📋 Next steps:
1) Review KRaft config: sudo grep -E 'process.roles|node.id|controller|listeners|advertised.listeners|log.dirs|metadata.log.dir' /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
2) Verify topics: /opt/kafka/bin/kafka-topics.sh --bootstrap-server eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092 --command-config /etc/kafka/client-web-universe-main-dev-coordinator.properties --list
✅ KRaft metadata layer setup completed successfully
Next steps:
1. Verify KRaft quorum status
2. Create Kafka topics
3. Configure Kafka Connect
[2026-02-05 06:33:49 UTC] USER=www-data EUID=0 PID=4145003 ACTION=fsop ARGS=mkdir -p /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev
[INFO] Saved metadata mode to: /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/kafka_metadata_mode
════════════════════════════════════════════════════════════════════
✅ Kafka Metadata Layer Setup Complete
Mode : kraft
Environment : web-universe-main-dev
State saved : /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/kafka_metadata_mode
KRaft cluster.id: uBayQf0-RSyd3l5SczIXzA
Verify quorum:
kafka-metadata-quorum.sh --bootstrap-server ... describe
════════════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Step 3 completed: 02-metadata.sh
[0;34m[INFO][0m 📦 Step 4/13: restart kafka related services...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 06:33:50 UTC] USER=www-data EUID=0 PID=4145068 ACTION=fsop ARGS=test -f /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 06:33:50 UTC] USER=www-data EUID=0 PID=4145077 ACTION=passthru ARGS=bash -c grep -E '^[[:space:]]*process\.roles=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties' | grep -Eq '(broker|controller)'
[INFO] 📋 Detected mode from server.properties: kraft
[2026-02-05 06:33:51 UTC] USER=www-data EUID=0 PID=4145276 ACTION=passthru ARGS=systemctl stop confluent-connect-web-universe-main-dev_coordinator.service
[2026-02-05 06:33:53 UTC] USER=www-data EUID=0 PID=4145410 ACTION=passthru ARGS=systemctl stop confluent-kafka-zk-web-universe-main-dev_coordinator.service
[2026-02-05 06:33:53 UTC] USER=www-data EUID=0 PID=4145487 ACTION=passthru ARGS=systemctl stop confluent-zookeeper-web-universe-main-dev_coordinator.service
Failed to stop confluent-zookeeper-web-universe-main-dev_coordinator.service: Unit confluent-zookeeper-web-universe-main-dev_coordinator.service not loaded.
[INFO] 🧹 Removing stale Kafka lock files...
[2026-02-05 06:33:56 UTC] USER=www-data EUID=0 PID=4145707 ACTION=fsop ARGS=rm -f /var/lib/kafka/web-universe-main-dev_coordinator-data/.lock
[INFO] 🧹 Cleaning up orphaned processes on Kafka ports...
[2026-02-05 06:33:56 UTC] USER=www-data EUID=0 PID=4145716 ACTION=passthru ARGS=bash -c
for port in 9092 9093 8083 2181; do
pids=$(lsof -ti tcp:$port 2>/dev/null || true)
if [[ -n "$pids" ]]; then
echo " Killing orphaned processes on port $port: $pids"
kill -9 $pids 2>/dev/null || true
sleep 1
fi
done
Killing orphaned processes on port 9092: 4094157
4095141
4095250
4095273
4095274
4108769
4113136
Killing orphaned processes on port 9093: 3750492
🚀 Restarting Kafka components…
[INFO] 🚀 starting confluent-kraft-web-universe-main-dev_coordinator.service…
[2026-02-05 06:34:02 UTC] USER=www-data EUID=0 PID=4146066 ACTION=passthru ARGS=systemctl restart confluent-kraft-web-universe-main-dev_coordinator.service
[INFO] 🚀 starting confluent-connect-web-universe-main-dev_coordinator.service…
[2026-02-05 06:34:03 UTC] USER=www-data EUID=0 PID=4146619 ACTION=passthru ARGS=systemctl restart confluent-connect-web-universe-main-dev_coordinator.service
[INFO] ⏳ Waiting for Kafka broker readiness (FQDN: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com, IP: 10.100.1.225) ...
[OK] ✅ Broker ready (attempt 1)
[OK] ✅ Port 9092 listening (Kafka Broker)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (1/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (2/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (3/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (4/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (5/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (6/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (7/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (8/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (9/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (10/40)
[OK] ✅ Port 8083 listening (Kafka Connect REST)
[INFO] ⏳ Waiting for Connect REST at https://eventbus-web-universe-main-dev-kafka-connect.fastorder.com:8083 …
[OK] ✅ Connect REST is up (attempt 1)
📋 Reconciling Connect internal topics…
[ok] connect-configs exists
[ok] connect-offsets exists
[ok] connect-status exists
═══════════════════════════════════════════════════════════════════
KAFKA SUMMARY
═══════════════════════════════════════════════════════════════════
Env: web-universe-main-dev Identifier: coordinator Mode: kraft
Broker Unit : confluent-kraft-web-universe-main-dev_coordinator.service (status: active)
Connect Unit: confluent-connect-web-universe-main-dev_coordinator.service (status: active)
Bootstrap : eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092
Connect URL : https://eventbus-web-universe-main-dev-kafka-connect.fastorder.com:8083
═══════════════════════════════════════════════════════════════════
[OK] ✅ All required services are up.
[0;32m[OK][0m ✅ Step 4 completed: 03-restart-kafka-related-services.sh
[0;34m[INFO][0m 📦 Step 5/13: checking services...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 06:35:53 UTC] USER=www-data EUID=0 PID=4155408 ACTION=fsop ARGS=test -f /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 06:35:54 UTC] USER=www-data EUID=0 PID=4155426 ACTION=passthru ARGS=bash -c grep -E '^[[:space:]]*process\.roles=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties' | grep -Eq '(broker|controller)'
[INFO] Detected mode from server.properties: kraft
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 1: Service status
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] confluent-kraft-web-universe-main-dev_coordinator.service status: active
[WARN] confluent-kafka-zk-web-universe-main-dev_coordinator.service present but should be stopped in KRaft
[WARN] confluent-zookeeper-web-universe-main-dev_coordinator.service present but not required in KRaft
[OK] confluent-connect-web-universe-main-dev_coordinator.service status: active
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 2: Port checks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Port 9092 listening (Kafka Broker)
[OK] ✅ Port 8083 listening (Kafka Connect REST)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 3: Broker readiness
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] Broker API responding (attempt 1)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 4: Kafka Connect REST
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] Connect REST responding (attempt 1)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Kafka Services Summary
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment : web-universe-main-dev
Identifier : coordinator
Mode : kraft
Broker Unit : confluent-kraft-web-universe-main-dev_coordinator.service (status: active)
Connect Unit: confluent-connect-web-universe-main-dev_coordinator.service (status: active)
Broker FQDN : eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092
Broker IP : eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092
Connect URL : https://eventbus-web-universe-main-dev-kafka-connect.fastorder.com:8083
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ All required services are reachable.
[INFO] Creating ACLs for Kafka Connect consumer groups...
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
[INFO] Creating ACLs for Connect internal topics...
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
[OK] ✅ Kafka Connect ACLs configured (deny-by-default mode)
[0;32m[OK][0m ✅ Step 5 completed: 04-checking-services.sh
[0;34m[INFO][0m 📦 Step 6/13: create audit topic...
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Creating Kafka Audit Topics
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Replication Factor: 1
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 🔍 AUDIT READINESS GATE - Preflight Checks
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m [Gate 1/5] Verifying DNS resolution...
[0;32m[OK][0m ✅ Broker DNS: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com -> 10.100.1.225
[0;32m[OK][0m ✅ Connect DNS: eventbus-web-universe-main-dev-kafka-connect.fastorder.com -> 10.100.1.75
[0;34m[INFO][0m [Gate 2/5] Verifying TLS handshake...
[0;32m[OK][0m ✅ TLS handshake: Broker certificate verified
[0;34m[INFO][0m [Gate 3/5] Verifying Kafka Connect REST API...
[0;32m[OK][0m ✅ Kafka Connect REST: Cluster ID = [2026-02-05 03:44:00 UTC] USER=www-data EUID=0 PID=3571696 ACTION=passthru ARGS=bash -c cat /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/kafka_kraft_cluster_id
uBayQf0-RSyd3l5SczIXzA
[0;34m[INFO][0m [Gate 4/5] Verifying required internal topics...
[0;32m[OK][0m ✅ Topic exists: connect-configs
[0;32m[OK][0m ✅ Topic exists: connect-offsets
[0;32m[OK][0m ✅ Topic exists: connect-status
[0;34m[INFO][0m [Gate 5/5] Verifying broker metadata access...
[0;32m[OK][0m ✅ Broker metadata: API versions accessible
[0;32m[OK][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ AUDIT READINESS GATE: ALL CHECKS PASSED
[0;32m[OK][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Waiting for Kafka to be ready...
[0;32m[OK][0m ✅ Kafka is ready
[0;34m[INFO][0m Creating audit topic: audit.events.web.universe.main.dev
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Error while executing topic command : Topic 'audit.events.web.universe.main.dev' already exists.
[2026-02-05 06:36:58,040] ERROR org.apache.kafka.common.errors.TopicExistsException: Topic 'audit.events.web.universe.main.dev' already exists.
(kafka.admin.TopicCommand$)
[0;32m[OK][0m ✅ Audit topic already exists: audit.events.web.universe.main.dev
[0;32m[OK][0m ✅ Topic verified: audit.events.web.universe.main.dev
Topic: audit.events.web.universe.main.dev TopicId: _h9pJC4FSPSnOavETTJe4Q PartitionCount: 3 ReplicationFactor: 1 Configs: compression.type=lz4,min.insync.replicas=1,cleanup.policy=delete,segment.bytes=1073741824,retention.ms=7776000000,message.timestamp.type=LogAppendTime,segment.ms=604800000
Topic: audit.events.web.universe.main.dev Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: audit.events.web.universe.main.dev Partition: 1 Leader: 1 Replicas: 1 Isr: 1
Topic: audit.events.web.universe.main.dev Partition: 2 Leader: 1 Replicas: 1 Isr: 1
[0;34m[INFO][0m Creating audit producer credentials...
Completed updating config for user audit-producer-web-universe-main-dev.
[0;32m[OK][0m ✅ Audit producer user created: audit-producer-web-universe-main-dev
[0;34m[INFO][0m Creating ACLs for audit producer...
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
[0;32m[OK][0m ✅ ACLs configured (producer: write-only, sinks: read-only, immutability: protected)
[0;34m[INFO][0m Storing audit producer credentials in AWS Secrets Manager...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/eventbus/web/universe/main/dev/kafka/audit/producer-X3Fpzs",
"Name": "fastorder/eventbus/web/universe/main/dev/kafka/audit/producer",
"VersionId": "b1b6da9c-f6bc-4254-9628-b77116d0777d"
}
[0;32m[OK][0m ✅ Credentials stored in: fastorder/eventbus/web/universe/main/dev/kafka/audit/producer
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Configuring S3 Sink for Audit Cold Storage
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
PRE audit/
[0;34m[INFO][0m S3 staging bucket exists: fastorder-audit-staging
[0;34m[INFO][0m Data flow: Kafka → fastorder-audit-staging → (replication) → fastorder-audit-immutable
[0;34m[INFO][0m Updating existing S3 sink connector...
{"name":"audit-s3-sink-web_universe_main_dev","config":{"connector.class":"io.confluent.connect.s3.S3SinkConnector","tasks.max":"1","topics":"audit.events.web.universe.main.dev","topics.dir":"audit/web-universe-main-dev","s3.bucket.name":"fastorder-audit-staging","s3.region":"me-central-1","s3.part.size":"5242880","flush.size":"1000","rotate.interval.ms":"3600000","rotate.schedule.interval.ms":"86400000","storage.class":"io.confluent.connect.s3.storage.S3Storage","format.class":"io.confluent.connect.s3.format.json.JsonFormat","partitioner.class":"io.confluent.connect.storage.partitioner.TimeBasedPartitioner","path.format":"'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH","partition.duration.ms":"3600000","locale":"en-US","timezone":"UTC","timestamp.extractor":"Record","key.converter":"org.apache.kafka.connect.json.JsonConverter","value.converter":"org.apache.kafka.connect.json.JsonConverter","key.converter.schemas.enable":"false","value.converter.schemas.enable":"false","behavior.on.null.values":"ignore","errors.tolerance":"all","errors.log.enable":"true","errors.log.include.messages":"true","name":"audit-s3-sink-web_universe_main_dev"},"tasks":[{"connector":"audit-s3-sink-web_universe_main_dev","task":0}],"type":"sink"}[0;32m[OK][0m ✅ S3 Sink connector configured for audit cold storage
[0;34m[INFO][0m Staging Bucket: fastorder-audit-staging (Kafka Connect writes here)
[0;34m[INFO][0m Immutable Bucket: fastorder-audit-immutable (via S3 Replication)
[0;34m[INFO][0m Path: audit/web-universe-main-dev/
[0;34m[INFO][0m Final Retention: WORM-enabled (Object Lock COMPLIANCE mode, 1-year)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Kafka Audit Topic Created Successfully
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Topic: audit.events.web.universe.main.dev
[0;34m[INFO][0m Partitions: 3
[0;34m[INFO][0m Replication Factor: 1
[0;34m[INFO][0m Retention: 90 days
[0;34m[INFO][0m Producer: audit-producer-web-universe-main-dev (write-only)
[0;34m[INFO][0m Application Integration:
[0;34m[INFO][0m - Use credentials from: fastorder/eventbus/web/universe/main/dev/kafka/audit/producer
[0;34m[INFO][0m - Connect to: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092
[0;34m[INFO][0m - Produce to: audit.events.web.universe.main.dev
[0;34m[INFO][0m - Security: SASL_SSL (SCRAM-SHA-512)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 📋 PCI-DSS Compliance Status
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m
[0;34m[INFO][0m ✅ Kafka Hot Storage: 90 days (meets PCI-DSS 3-month immediate access)
[0;34m[INFO][0m ✅ ACL Authorization: deny-by-default (allow.everyone.if.no.acl.found=false)
[0;34m[INFO][0m ✅ Immutability: DENY DELETE/ALTER on audit topic
[0;34m[INFO][0m ✅ S3 Cold Storage: fastorder-audit-immutable (Object Lock COMPLIANCE, 1-year)
[0;34m[INFO][0m
[0;34m[INFO][0m S3 Audit Storage:
[0;34m[INFO][0m Bucket: s3://fastorder-audit-staging
[0;34m[INFO][0m Path: audit/web-universe-main-dev/
[0;34m[INFO][0m Object Lock: COMPLIANCE mode, 1-year retention
[0;34m[INFO][0m Immutability: Objects cannot be deleted or modified for 1 year
[0;34m[INFO][0m
[0;34m[INFO][0m Verify compliance with:
[0;34m[INFO][0m bash 04-eventbus/engine/kafka/steps/11-audit-compliance-check.sh
[0;34m[INFO][0m
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 📋 AWS Roles Anywhere - Credential Refresh Setup
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m
[0;34m[INFO][0m For S3 sink to write to S3, Kafka Connect needs AWS credentials.
[0;34m[INFO][0m Use IAM Roles Anywhere with systemd timer for automatic refresh.
[0;34m[INFO][0m
[0;34m[INFO][0m Required files:
[0;34m[INFO][0m Certificate: /etc/fastorder/rolesanywhere/client-bundle.crt
[0;34m[INFO][0m Private Key: /etc/fastorder/rolesanywhere/client.key
[0;34m[INFO][0m Helper: /usr/local/bin/aws_signing_helper
[0;34m[INFO][0m
[0;34m[INFO][0m Systemd timer: kafka-aws-credential-refresh.timer
[0;34m[INFO][0m Runs every 30 minutes to refresh credentials to /var/lib/kafka/.aws/credentials
[0;34m[INFO][0m
[0;34m[INFO][0m Verify timer is active:
[0;34m[INFO][0m systemctl status kafka-aws-credential-refresh.timer
[0;34m[INFO][0m
[0;34m[INFO][0m Documentation: https://docs.aws.amazon.com/rolesanywhere/latest/userguide/
[0;32m[OK][0m ✅ Step 6 completed: 05-create-audit-topic.sh
[0;34m[INFO][0m 📦 Step 7/13: setup backups...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Kafka Backup Configuration
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Environment: web-universe-main-dev
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[0;34m[INFO][0m 1️⃣ Creating S3 bucket for Kafka backups...
make_bucket failed: s3://fastorder-kafka-backups-web-universe-main-dev An error occurred (AccessDenied) when calling the CreateBucket operation: User: arn:aws:iam::464621692046:user/fo-dev is not authorized to perform: s3:CreateBucket on resource: "arn:aws:s3:::fastorder-kafka-backups-web-universe-main-dev" because no identity-based policy allows the s3:CreateBucket action
An error occurred (NoSuchBucket) when calling the PutBucketVersioning operation: The specified bucket does not exist
Parameter validation failed:
Unknown parameter in LifecycleConfiguration.Rules[0]: "Id", must be one of: Expiration, ID, Prefix, Filter, Status, Transitions, NoncurrentVersionTransitions, NoncurrentVersionExpiration, AbortIncompleteMultipartUpload
[0;32m[OK][0m ✅ S3 bucket created: fastorder-kafka-backups-web-universe-main-dev
[0;34m[INFO][0m 2️⃣ Creating local backup directory...
[2026-02-05 06:37:53 UTC] USER=www-data EUID=0 PID=4163074 ACTION=fsop ARGS=mkdir -p /var/backups/kafka/web-universe-main-dev/topics
[2026-02-05 06:37:53 UTC] USER=www-data EUID=0 PID=4163092 ACTION=fsop ARGS=mkdir -p /var/log/kafka/backups
[2026-02-05 06:37:53 UTC] USER=www-data EUID=0 PID=4163101 ACTION=fsop ARGS=chown -R kafka:kafka /var/backups/kafka/web-universe-main-dev
[2026-02-05 06:37:53 UTC] USER=www-data EUID=0 PID=4163110 ACTION=fsop ARGS=chown -R kafka:kafka /var/log/kafka/backups
[2026-02-05 06:37:53 UTC] USER=www-data EUID=0 PID=4163122 ACTION=fsop ARGS=chmod 750 /var/backups/kafka/web-universe-main-dev
[0;32m[OK][0m ✅ Local backup directory created
[0;34m[INFO][0m 3️⃣ Creating topic backup script...
[2026-02-05 06:37:53 UTC] USER=www-data EUID=0 PID=4163145 ACTION=fsop ARGS=sed -i s|__ENV_ID__|web-universe-main-dev|g /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[2026-02-05 06:37:53 UTC] USER=www-data EUID=0 PID=4163155 ACTION=fsop ARGS=sed -i s|__KAFKA_BROKER__|eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092|g /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[2026-02-05 06:37:53 UTC] USER=www-data EUID=0 PID=4163166 ACTION=fsop ARGS=sed -i s|__BACKUP_DIR__|/var/backups/kafka/web-universe-main-dev|g /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[2026-02-05 06:37:53 UTC] USER=www-data EUID=0 PID=4163175 ACTION=fsop ARGS=sed -i s|__S3_BUCKET__|fastorder-kafka-backups-web-universe-main-dev|g /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[2026-02-05 06:37:53 UTC] USER=www-data EUID=0 PID=4163184 ACTION=fsop ARGS=sed -i s|__S3_REGION__|me-central-1|g /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[2026-02-05 06:37:53 UTC] USER=www-data EUID=0 PID=4163193 ACTION=fsop ARGS=chmod 750 /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[2026-02-05 06:37:53 UTC] USER=www-data EUID=0 PID=4163202 ACTION=fsop ARGS=chown root:kafka /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[0;32m[OK][0m ✅ Backup script created: /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[0;34m[INFO][0m 4️⃣ Setting up cron jobs for automated backups...
[2026-02-05 06:37:54 UTC] USER=www-data EUID=0 PID=4163220 ACTION=fsop ARGS=chmod 644 /etc/cron.d/kafka-backups-web-universe-main-dev
[0;32m[OK][0m ✅ Cron job configured: Daily backups at 2:00 AM
[0;34m[INFO][0m 5️⃣ Creating restore documentation...
[2026-02-05 06:37:54 UTC] USER=www-data EUID=0 PID=4163241 ACTION=fsop ARGS=sed -i s|__S3_BUCKET__|fastorder-kafka-backups-web-universe-main-dev|g /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[2026-02-05 06:37:54 UTC] USER=www-data EUID=0 PID=4163250 ACTION=fsop ARGS=sed -i s|__S3_REGION__|me-central-1|g /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[2026-02-05 06:37:54 UTC] USER=www-data EUID=0 PID=4163259 ACTION=fsop ARGS=sed -i s|__KAFKA_BROKER__|eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com|g /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[2026-02-05 06:37:54 UTC] USER=www-data EUID=0 PID=4163268 ACTION=fsop ARGS=chmod 644 /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[2026-02-05 06:37:54 UTC] USER=www-data EUID=0 PID=4163277 ACTION=fsop ARGS=chown kafka:kafka /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[0;32m[OK][0m ✅ Restore documentation created: /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Kafka Backup Configured
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m S3 Bucket: fastorder-kafka-backups-web-universe-main-dev
[0;34m[INFO][0m Region: me-central-1
[0;34m[INFO][0m Local backup dir: /var/backups/kafka/web-universe-main-dev
[0;34m[INFO][0m Schedule: Daily at 2:00 AM
[0;34m[INFO][0m Script: /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[0;34m[INFO][0m Restore docs: /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[1;33m[WARN][0m ⚠️ Note: This backs up Kafka metadata only (topics, configs, offsets)
[1;33m[WARN][0m For full message data backup, configure Kafka Connect S3 Sink
[0;32m[OK][0m ✅ Step 7 completed: 06-setup-backups.sh
[0;34m[INFO][0m 📦 Step 8/13: monitoring setup...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Kafka Monitoring Integration for web-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-web-universe-main-dev
[1;32m[OK][0m Observability cell endpoints registered for web-universe-main-dev
[1;32m[OK][0m ✓ Observability cell is ready
[INFO] 2️⃣ Setting up Kafka JMX exporter integration...
[INFO] JMX Exporter port calculated for web-universe-main-dev: 9362 (offset: 54)
[INFO] Checking observability cell readiness: obs-web-universe-main-dev
[1;32m[OK][0m Observability cell endpoints registered for web-universe-main-dev
[INFO] Setting up Kafka JMX exporter for web-universe-main-dev
[INFO] JMX Prometheus Java Agent already exists at /opt/kafka/libs/jmx_prometheus_javaagent.jar
[2026-02-05 06:37:54 UTC] USER=www-data EUID=0 PID=4163334 ACTION=passthru ARGS=mv /tmp/jmx_exporter.yml /opt/kafka/config/jmx_exporter.yml
[2026-02-05 06:37:54 UTC] USER=www-data EUID=0 PID=4163349 ACTION=passthru ARGS=chmod 644 /opt/kafka/config/jmx_exporter.yml
[1;32m[OK][0m JMX exporter configuration created at /opt/kafka/config/jmx_exporter.yml
[1;32m[OK][0m JMX exporter configuration created
[INFO] Configuring Kafka systemd services to use JMX exporter...
[2026-02-05 06:37:54 UTC] USER=www-data EUID=0 PID=4163396 ACTION=fsop ARGS=test -f /etc/systemd/system/[2026-02-05
[INFO] All Kafka services already configured with JMX exporter
[1;32m[OK][0m Kafka JMX exporter integration complete
[INFO] Metrics endpoint: http://142.93.238.16:9362/metrics
[INFO] Prometheus will automatically scrape: https://metrics-web-universe-main-dev.fastorder.com:9090
[INFO] View dashboards at: https://dashboards-web-universe-main-dev.fastorder.com
[1;32m[OK][0m ✓ Kafka JMX exporter integration complete
[INFO] Configuring KAFKA_OPTS environment variable for kafka user...
[2026-02-05 06:37:55 UTC] USER=www-data EUID=0 PID=4163433 ACTION=passthru ARGS=grep -q KAFKA_OPTS.*javaagent.*jmx_prometheus_javaagent.*=9362: /home/kafka/.bashrc
[2026-02-05 06:37:55 UTC] USER=www-data EUID=0 PID=4163456 ACTION=passthru ARGS=sed -i /export KAFKA_OPTS=.*jmx_prometheus_javaagent/d /home/kafka/.bashrc
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[1;32m[OK][0m ✓ KAFKA_OPTS configured in /home/kafka/.bashrc (port 9362)
[INFO] 2.5️⃣ Enabling JMX exporter in Kafka systemd service...
[2026-02-05 06:37:55 UTC] USER=www-data EUID=0 PID=4163492 ACTION=passthru ARGS=grep -q javaagent.*jmx_prometheus_javaagent /etc/systemd/system/confluent-kraft-web-universe-main-dev_coordinator.service
[INFO] Updating confluent-kraft-web-universe-main-dev_coordinator.service to enable JMX exporter...
[2026-02-05 06:37:55 UTC] USER=www-data EUID=0 PID=4163541 ACTION=passthru ARGS=sed -i s|^Environment=KAFKA_OPTS=.*|Environment=KAFKA_OPTS=-javaagent:/opt/kafka/libs/jmx_prometheus_javaagent.jar=9362:/opt/kafka/config/jmx_exporter.yml| /etc/systemd/system/confluent-kraft-web-universe-main-dev_coordinator.service
[1;32m[OK][0m ✓ Updated confluent-kraft-web-universe-main-dev_coordinator.service
[INFO] Reloading systemd daemon and restarting Kafka services...
[2026-02-05 06:37:55 UTC] USER=www-data EUID=0 PID=4163565 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 06:37:56 UTC] USER=www-data EUID=0 PID=4163633 ACTION=passthru ARGS=systemctl is-active --quiet confluent-kraft-web-universe-main-dev_coordinator
[INFO] Restarting confluent-kraft-web-universe-main-dev_coordinator...
[2026-02-05 06:37:56 UTC] USER=www-data EUID=0 PID=4163654 ACTION=passthru ARGS=systemctl restart confluent-kraft-web-universe-main-dev_coordinator
[2026-02-05 06:38:01 UTC] USER=www-data EUID=0 PID=4164217 ACTION=passthru ARGS=systemctl is-active --quiet confluent-kraft-web-universe-main-dev_coordinator
[1;32m[OK][0m ✓ confluent-kraft-web-universe-main-dev_coordinator restarted successfully
[1;32m[OK][0m ✓ JMX exporter enabled in Kafka systemd services
[INFO] 2.6️⃣ Configuring Prometheus to scrape Kafka metrics...
[2026-02-05 06:38:01 UTC] USER=www-data EUID=0 PID=4164242 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
[INFO] Adding Kafka scrape target to Prometheus configuration...
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[2026-02-05 06:38:01 UTC] USER=www-data EUID=0 PID=4164275 ACTION=passthru ARGS=sed -i /# Prometheus self-monitoring/r /tmp/prometheus_kafka_add.yml /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
[ERROR] Invalid Prometheus configuration - rolling back
[2026-02-05 06:38:01 UTC] USER=www-data EUID=0 PID=4164330 ACTION=passthru ARGS=sed -i /job_name: 'kafka'/,+6d /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
[2026-02-05 06:38:01 UTC] USER=www-data EUID=0 PID=4164351 ACTION=fsop ARGS=rm -f /tmp/prometheus_kafka_add.yml
[INFO] 3️⃣ Registering Kafka nodes to monitoring database...
[INFO] Detected Kafka version: 3.9.1
[INFO] Registering Kafka Broker to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Kafka Broker
[INFO] Identifier: web-universe-main-dev-broker-01
[INFO] Identifier Parent: cluster
[INFO] IP: 142.93.238.16
[INFO] Port: 9092
[INFO] FQDN: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 0fe22eef-a876-408e-9099-f79ee8d192b7
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[1;32m[OK][0m ✓ Kafka broker registered
[INFO] Registering Kafka Connect to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Kafka Connect
[INFO] Identifier: web-universe-main-dev-connect-01
[INFO] Identifier Parent: cluster
[INFO] IP: 142.93.238.16
[INFO] Port: 8083
[INFO] FQDN: eventbus-web-universe-main-dev-kafka-connect.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 269c6f12-e045-4268-8bc4-73c5e936d212
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[1;32m[OK][0m ✓ Kafka Connect registered
[INFO] Schema Registry not running, skipping registration
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Kafka Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Metrics: http://localhost:9362/metrics
[INFO] Prometheus: https://metrics-web-universe-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-web-universe-main-dev.fastorder.com
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Step 8 completed: 10-monitoring-setup.sh
[0;34m[INFO][0m 📦 Step 9/13: audit compliance check...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[1m KAFKA AUDIT COMPLIANCE DASHBOARD - PCI-DSS Verification[0m
[1m Environment: web-universe-main-dev[0m
[1m Timestamp: 2026-02-05 06:38:07 UTC[0m
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[1m[1/5] Kafka Deny-by-Default ACL Posture[0m
Requirement: allow.everyone.if.no.acl.found=false
[0;32mPASS[0m Deny-by-default is ENABLED (allow.everyone.if.no.acl.found=false)
[1m[2/5] Audit Topic Hot Retention (90 days)[0m
Requirement: retention.ms >= 7776000000 (90 days)
[0;32mPASS[0m Retention is 90 days (7776000000 ms)
[1m[3/5] Kafka Connect S3 Sink Status[0m
Requirement: Connector and all tasks RUNNING
[0;31mFAIL[0m Connector RUNNING but tasks FAILED: FAILED
[1m[4/5] S3 Freshness Evidence[0m
Requirement: Newest object < 120 minutes old
[1;33mWARN[0m No objects found in s3://fastorder-audit-immutable/audit/web-universe-main-dev/
This may be normal if no audit events have been generated yet
[1m[5/5] S3 Object Lock Immutability[0m
Requirement: COMPLIANCE mode with 1-year retention
[0;31mFAIL[0m Cannot verify Object Lock configuration - access denied
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[1m COMPLIANCE SUMMARY[0m
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;31m[1mCOMPLIANCE ISSUES DETECTED[0m
Passed: 2/5
Failed: 3/5
Review failed checks above and remediate.
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;32m[OK][0m ✅ Step 9 completed: 11-audit-compliance-check.sh
[0;34m[INFO][0m 📦 Step 10/13: audit canary test...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[1m KAFKA AUDIT CANARY TEST - End-to-End Verification[0m
[1m Environment: web-universe-main-dev[0m
[1m Canary ID: canary-1770273499-4165843[0m
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[1m[Step 1/4] Producing canary audit event to Kafka[0m
Topic: audit.events.web.universe.main.dev
Event: canary-1770273499-4165843
[0;31mFailed to produce event[0m
[1;33m(Topic may not exist yet - normal during initial setup)[0m
[0;32m[OK][0m ✅ Step 10 completed: 12-audit-canary-test.sh
[0;34m[INFO][0m 📦 Step 11/13: setup audit s3 staging...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
════════════════════════════════════════════════════════════════════════
Kafka Audit S3 Staging + Replication Setup
════════════════════════════════════════════════════════════════════════
Staging Bucket: fastorder-audit-staging
Immutable Bucket: fastorder-audit-immutable
Region: me-central-1
Environment: --auto
════════════════════════════════════════════════════════════════════════
[INFO] 1️⃣ Checking AWS credentials...
[WARN] No AWS credentials found - skipping S3 staging setup
To configure S3 audit storage later, add credentials to /home/ab/.aws/credentials:
[admin]
aws_access_key_id = AKIA...
aws_secret_access_key = ...
Then run: AWS_PROFILE=admin /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps/13-setup-audit-s3-staging.sh --auto
[0;32m[OK][0m ✅ Step 11 completed: 13-setup-audit-s3-staging.sh
[0;34m[INFO][0m 📦 Step 12/13: install ksqldb...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ksqlDB Installation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Identifier: --auto
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
VM_IP: 10.100.1.242
FQDN: eventbus-web-universe-main-dev-ksqldb---auto.fastorder.com
📦 Step 1: Checking Confluent Platform installation...
✅ ksqlDB already installed (version: )
📁 Step 2: Creating directories...
[2026-02-05 06:38:25 UTC] USER=www-data EUID=0 PID=4166483 ACTION=fsop ARGS=mkdir -p /var/lib/ksqldb/web-universe-main-dev/--auto
[2026-02-05 06:38:25 UTC] USER=www-data EUID=0 PID=4166507 ACTION=fsop ARGS=mkdir -p /var/log/ksqldb/web-universe-main-dev/--auto
[2026-02-05 06:38:25 UTC] USER=www-data EUID=0 PID=4166528 ACTION=fsop ARGS=mkdir -p /etc/ksqldb/web-universe-main-dev/--auto
[2026-02-05 06:38:25 UTC] USER=www-data EUID=0 PID=4166549 ACTION=fsop ARGS=chown -R kafka:kafka /var/lib/ksqldb/web-universe-main-dev/--auto /var/log/ksqldb/web-universe-main-dev/--auto /etc/ksqldb/web-universe-main-dev/--auto
✅ Directories created
⚙️ Step 3: Generating ksqlDB configuration...
[2026-02-05 06:38:25 UTC] USER=www-data EUID=0 PID=4166574 ACTION=fsop ARGS=mv /tmp/ksql-server-web-universe-main-dev.properties /etc/ksqldb/web-universe-main-dev/--auto/ksql-server.properties
[2026-02-05 06:38:25 UTC] USER=www-data EUID=0 PID=4166597 ACTION=fsop ARGS=chown kafka:kafka /etc/ksqldb/web-universe-main-dev/--auto/ksql-server.properties
[2026-02-05 06:38:25 UTC] USER=www-data EUID=0 PID=4166618 ACTION=fsop ARGS=chmod 640 /etc/ksqldb/web-universe-main-dev/--auto/ksql-server.properties
✅ Configuration generated: /etc/ksqldb/web-universe-main-dev/--auto/ksql-server.properties
🔧 Step 4: Creating systemd service...
[2026-02-05 06:38:25 UTC] USER=www-data EUID=0 PID=4166640 ACTION=fsop ARGS=mv /tmp/ksqldb-web-universe-main-dev---auto.service /etc/systemd/system/ksqldb-web-universe-main-dev---auto.service
[2026-02-05 06:38:25 UTC] USER=www-data EUID=0 PID=4166661 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 06:38:26 UTC] USER=www-data EUID=0 PID=4166739 ACTION=passthru ARGS=systemctl enable ksqldb-web-universe-main-dev---auto.service
✅ Systemd service created: ksqldb-web-universe-main-dev---auto.service
🚀 Step 5: Starting ksqlDB service...
🔍 Checking Kafka broker connectivity...
✅ Kafka broker is accessible
[2026-02-05 06:38:26 UTC] USER=www-data EUID=0 PID=4166813 ACTION=passthru ARGS=systemctl start ksqldb-web-universe-main-dev---auto.service
✅ ksqlDB service started
⏳ Waiting for ksqlDB to be ready...
..............................
🔍 Step 6: Verifying installation...
📊 Service Status:
[2026-02-05 06:39:28 UTC] USER=www-data EUID=0 PID=4168979 ACTION=passthru ARGS=systemctl status ksqldb-web-universe-main-dev---auto.service --no-pager -l
📊 ksqlDB Info:
⚠️ ksqlDB not responding yet (may still be starting)
📡 Step 7: Registering ksqlDB to Observability API...
🔄 Registering ksqlDB node to observability dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: ksqlDB
[INFO] Identifier: web-universe-main-dev-ksqldb---auto
[INFO] Identifier Parent: eventbus
[INFO] IP: 10.100.1.242
[INFO] Port: 8088
[INFO] FQDN: eventbus-web-universe-main-dev-ksqldb---auto.fastorder.com
[INFO] Status: starting
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 959bfeea-5527-4a0f-84cb-9c8e8a9d7811
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
✅ ksqlDB registered successfully
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ksqlDB Installation Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Service: ksqldb-web-universe-main-dev---auto
VM_IP: 10.100.1.242
FQDN: eventbus-web-universe-main-dev-ksqldb---auto.fastorder.com
Port: 8088
Config: /etc/ksqldb/web-universe-main-dev/--auto/ksql-server.properties
Data: /var/lib/ksqldb/web-universe-main-dev/--auto
Logs: /var/log/ksqldb/web-universe-main-dev/--auto
Dashboard:
https://skeleton.dev.fastorder.com/dashboard/monitoring/environment2/<env-id>/service/ksqldb
CLI Access (with SSL):
ksql --ssl https://eventbus-web-universe-main-dev-ksqldb---auto.fastorder.com:8088
REST API (HTTPS):
curl -k https://eventbus-web-universe-main-dev-ksqldb---auto.fastorder.com:8088/info
curl -k https://eventbus-web-universe-main-dev-ksqldb---auto.fastorder.com:8088/ksql -H 'Content-Type: application/vnd.ksql.v1+json' -d '{"ksql": "SHOW STREAMS;"}'
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Step 12 completed: 20-install-ksqldb.sh
[0;34m[INFO][0m 📦 Step 13/13: update www data certs...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
📋 Post-Kafka Setup: Updating www-data Kafka certificates...
Environment: web-universe-main-dev
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
✓ Kafka certificates found
✓ www-data user exists
[2026-02-05 06:39:29 UTC] USER=www-data EUID=0 PID=4169051 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:39:29 UTC] USER=www-data EUID=0 PID=4169063 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:39:29 UTC] USER=www-data EUID=0 PID=4169081 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 06:39:29 UTC] USER=www-data EUID=0 PID=4169094 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 06:39:29 UTC] USER=www-data EUID=0 PID=4169107 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 06:39:29 UTC] USER=www-data EUID=0 PID=4169116 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:39:29 UTC] USER=www-data EUID=0 PID=4169125 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:39:29 UTC] USER=www-data EUID=0 PID=4169134 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:39:29 UTC] USER=www-data EUID=0 PID=4169143 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks created for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✓ Post-Kafka setup complete
[0;32m[OK][0m ✅ Step 13 completed: 99-update-www-data-certs.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Kafka setup completed successfully!
[0;32m[OK][0m Executed all 13 steps
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Service: web
[0;34m[INFO][0m Zone: universe
[0;34m[INFO][0m Branch: main
[0;34m[INFO][0m Env: dev
[0;34m[INFO][0m Registering Kafka nodes via API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Kafka
[INFO] Identifier: web-universe-main-dev_coordinator
[INFO] Identifier Parent: coordinator
[INFO] IP: 10.100.1.225
[INFO] Port: 9092
[INFO] FQDN: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 556513cd-577f-4835-837d-7f8a97c24183
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Kafka Connect
[INFO] Identifier: web-universe-main-dev_coordinator
[INFO] Identifier Parent: coordinator
[INFO] IP: 10.100.1.75
[INFO] Port: 8083
[INFO] FQDN: eventbus-web-universe-main-dev-kafka-connect.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 832e2b3d-94f8-4caf-9464-57bb9914f0a8
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[0;32m[OK][0m ✔ Kafka node registration completed
[0;34m[INFO][0m Setting up Kafka observability integration...
[0;34m[INFO][0m Checking observability cell readiness: obs-web-universe-main-dev
[0;32m[OK][0m Observability cell endpoints registered for web-universe-main-dev
[0;34m[INFO][0m Observability cell verified for web-universe-main-dev
[0;34m[INFO][0m Monitoring will be configured after Kafka deployment (step 10-monitoring-setup.sh)
[0;34m[INFO][0m Cleaning up temporary files...
[INFO] Starting cleanup of temporary files...
[INFO] Cleaning up SSL temp files for web-universe-main-dev...
[INFO] Cleaning up old provisioning logs...
[INFO] Cleaning up old configuration backups...
[0;32m[OK][0m ✔ Cleanup completed
[0;32m✓[0m ✅ Event bus infrastructure (kafka) setup completed successfully
[0;34m[INFO][0m Using database engine from DB_ENGINE environment variable: postgresql
[0;34m[INFO][0m Cleaning up any existing locks...
[0;32m[1mStarting database engine: postgresql[0m
[1;33m═══════════════════════════════════════════════[0m
[0;34m[INFO][0m Using environment from web interface: web-universe-main-dev
[0;32m[2026-02-05 06:39:31][0m Using web-provided environment: web-universe-main-dev
[0;32m[2026-02-05 06:39:31][0m Service: web, Zone: universe, Branch: main, Env: dev
[0;32m✓[0m Environment initialized successfully (mode: general)
[0;34m[INFO][0m Checking observability cell readiness: obs-web-universe-main-dev
[1;32m[OK][0m Observability cell endpoints registered for web-universe-main-dev
[0;34m[INFO][0m Observability cell verified for web-universe-main-dev
[0;34m[INFO][0m Monitoring will be configured after PostgreSQL deployment (step 10-monitoring-setup.sh)
[0;34m[INFO][0m Citus mode ENABLED
[0;34m[INFO][0m → Coordinator + 1 worker(s) + 0 standby node(s) per worker
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up coordinator (Citus control plane)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-02-05 06:39:32 UTC] USER=unknown EUID=33 PID=4169474 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 06:39:32 UTC] USER=unknown EUID=33 PID=4169481 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 06:39:32 UTC] USER=unknown EUID=33 PID=4169488 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-02-05 06:39:32 UTC] USER=unknown EUID=33 PID=4169495 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 06:39:32 UTC] USER=unknown EUID=33 PID=4169502 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-02-05 06:39:32 UTC] USER=unknown EUID=33 PID=4169509 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for web in universe-dev...
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=da95d2d5-7a04-4279-ae1c-c482185a88f1 JOB_UUID=32abb742-02ad-4ac9-8236-502afdefc367
[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=da95d2d5-7a04-4279-ae1c-c482185a88f1)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.54
[0;34m[INFO][0m Primary hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[0;34m[INFO][0m Adding /etc/hosts entries for coordinator...
[0;34m[INFO][0m 1. db-web-universe-main-dev-postgresql.fastorder.com → 10.100.1.54 (primary/short)
[0;34m[INFO][0m 2. db-web-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.54 (compatibility)
[0;34m[INFO][0m ✅ db-web-universe-main-dev-postgresql.fastorder.com already exists with correct IP
[0;34m[INFO][0m ✅ db-web-universe-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
10.100.1.54 db-web-universe-main-dev-postgresql.fastorder.com
10.100.1.54 db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=da95d2d5-7a04-4279-ae1c-c482185a88f1)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe (FastOrder Universe)
Branch: main
Env: dev
Node: coordinator
Primary CN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Alt CN: web-universe-main-dev.fastorder.com
Internal IP: 10.100.1.54 (from topology: db-coordinator-postgresql)
Shared IP: 142.93.238.16 (legacy/public, also included in SANs)
Coordinator variants:
- db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
- db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 06:39:35 UTC] USER=www-data EUID=0 PID=4169664 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator and /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:39:35 UTC] USER=www-data EUID=0 PID=4169673 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-02-05 06:39:35 UTC] USER=www-data EUID=0 PID=4169683 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-4169625
[2026-02-05 06:39:35 UTC] USER=www-data EUID=0 PID=4169701 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-4169625/ra_root.key
[2026-02-05 06:39:35 UTC] USER=www-data EUID=0 PID=4169710 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4169625/ra_root.crt
[2026-02-05 06:39:35 UTC] USER=www-data EUID=0 PID=4169719 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4169625/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[2026-02-05 06:39:36 UTC] USER=www-data EUID=0 PID=4169776 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4169625/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 06:39:36 UTC] USER=www-data EUID=0 PID=4169785 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4169625/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 06:39:36 UTC] USER=www-data EUID=0 PID=4169794 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-02-05 06:39:36 UTC] USER=www-data EUID=0 PID=4169803 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4169625/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:36 UTC] USER=www-data EUID=0 PID=4169812 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:36 UTC] USER=www-data EUID=0 PID=4169830 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 06:39:36 UTC] USER=www-data EUID=0 PID=4169841 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 06:39:37 UTC] USER=www-data EUID=0 PID=4169850 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 06:39:37 UTC] USER=www-data EUID=0 PID=4169859 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 06:39:37 UTC] USER=www-data EUID=0 PID=4169868 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 06:39:37 UTC] USER=www-data EUID=0 PID=4169877 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:39:37 UTC] USER=www-data EUID=0 PID=4169886 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔍 Verifying certificate...
Certificate details:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Node: coordinator
Primary CN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-coordinator.service
3. Test SSL connection:
psql "host=db-web-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: postgres
Identifier: coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: coordinator
User (CN): postgres
Hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 06:39:37 UTC] USER=www-data EUID=0 PID=4169944 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-02-05 06:39:37 UTC] USER=www-data EUID=0 PID=4169958 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 06:39:37 UTC] USER=www-data EUID=0 PID=4169967 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-02-05 06:39:37 UTC] USER=www-data EUID=0 PID=4169976 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 06:39:37 UTC] USER=www-data EUID=0 PID=4169985 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170010 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170021 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170035 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170059 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170071 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170080 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170089 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170098 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170107 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170116 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170125 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170135 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170146 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170155 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170169 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170189 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170211 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:38 UTC] USER=www-data EUID=0 PID=4170235 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170264 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170277 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170289 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170298 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170307 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170316 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170325 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170334 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170343 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170352 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170361 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170370 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170382 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170401 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170413 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170422 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170431 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170440 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170449 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170458 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170467 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170477 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170488 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170506 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170516 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:39 UTC] USER=www-data EUID=0 PID=4170527 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170536 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170545 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170561 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170580 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170589 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170603 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170613 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170622 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170631 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170640 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170653 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170665 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170675 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170684 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170697 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170715 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170724 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 06:39:40 UTC] USER=www-data EUID=0 PID=4170734 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:41 UTC] USER=www-data EUID=0 PID=4170743 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:39:41 UTC] USER=www-data EUID=0 PID=4170755 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 06:39:41 UTC] USER=www-data EUID=0 PID=4170769 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 06:39:41 UTC] USER=www-data EUID=0 PID=4170778 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 06:39:41 UTC] USER=www-data EUID=0 PID=4170788 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:39:41 UTC] USER=www-data EUID=0 PID=4170798 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:39:41 UTC] USER=www-data EUID=0 PID=4170807 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:39:41 UTC] USER=www-data EUID=0 PID=4170816 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
✅ Symlinked client-cert.pem
[2026-02-05 06:39:41 UTC] USER=www-data EUID=0 PID=4170834 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 06:39:41 UTC] USER=www-data EUID=0 PID=4170843 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:39:41 UTC] USER=www-data EUID=0 PID=4170852 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:39:41 UTC] USER=www-data EUID=0 PID=4170861 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:39:41 UTC] USER=www-data EUID=0 PID=4170870 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: postgres
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: postgres
Identifier: coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: coordinator
User (CN): postgres
Hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 06:39:41 UTC] USER=www-data EUID=0 PID=4170922 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-02-05 06:39:42 UTC] USER=www-data EUID=0 PID=4170931 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-02-05 06:39:42 UTC] USER=www-data EUID=0 PID=4170952 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:39:42 UTC] USER=www-data EUID=0 PID=4170994 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:39:42 UTC] USER=www-data EUID=0 PID=4171003 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:39:42 UTC] USER=www-data EUID=0 PID=4171012 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 06:39:42 UTC] USER=www-data EUID=0 PID=4171022 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 06:39:42 UTC] USER=www-data EUID=0 PID=4171031 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171040 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171049 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171063 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171079 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171089 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171098 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171107 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171116 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171125 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171134 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171143 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171162 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171171 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171180 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171198 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171227 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171237 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171246 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171255 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171264 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171273 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171282 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171291 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171301 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171310 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171319 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171328 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 06:39:43 UTC] USER=www-data EUID=0 PID=4171338 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171348 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171357 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171366 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171375 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171386 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171395 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171404 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171413 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171422 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171431 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171440 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171449 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171459 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171478 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171487 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171496 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171506 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171526 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171537 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171546 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171555 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171564 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_der.key
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171573 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres_pk8.der
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171583 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171593 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171602 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171611 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171620 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171629 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171639 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key
[2026-02-05 06:39:44 UTC] USER=www-data EUID=0 PID=4171654 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
[2026-02-05 06:39:45 UTC] USER=www-data EUID=0 PID=4171667 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:45 UTC] USER=www-data EUID=0 PID=4171676 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:39:45 UTC] USER=www-data EUID=0 PID=4171685 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-02-05 06:39:45 UTC] USER=www-data EUID=0 PID=4171725 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:39:45 UTC] USER=www-data EUID=0 PID=4171736 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:39:45 UTC] USER=www-data EUID=0 PID=4171745 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
✅ Symlinked ca.pem
[2026-02-05 06:39:45 UTC] USER=www-data EUID=0 PID=4171765 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 06:39:45 UTC] USER=www-data EUID=0 PID=4171774 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 06:39:45 UTC] USER=www-data EUID=0 PID=4171783 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:39:45 UTC] USER=www-data EUID=0 PID=4171792 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:39:45 UTC] USER=www-data EUID=0 PID=4171804 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:39:45 UTC] USER=www-data EUID=0 PID=4171818 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: postgres
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres
[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=da95d2d5-7a04-4279-ae1c-c482185a88f1)
[INFO] 📦 02 setup pg instance...
[0;34m[DEADLOCK-PREVENTION][0m Deadlock prevention library loaded
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Using existing db-coordinator-postgresql environment: db-web-universe-main-dev-postgresql-coordinator.fastorder.com (10.100.1.54)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.54
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m Data dir: /data/postgresql/17/web-universe-main-dev/coordinator
[0;34m[INFO][0m Port: 5432
[0;34m[INFO][0m Hostname: db-web-universe-main-dev-postgresql-coordinator
[2026-02-05 06:39:46 UTC] USER=www-data EUID=0 PID=4171928 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:39:47 UTC] USER=www-data EUID=0 PID=4171950 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:39:47 UTC] USER=www-data EUID=0 PID=4171976 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:39:47 UTC] USER=www-data EUID=0 PID=4171997 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[1;33m[WARN][0m Server certificate not found at /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[0;34m[INFO][0m Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe (FastOrder Universe)
Branch: main
Env: dev
Node: coordinator
Primary CN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Alt CN: web-universe-main-dev.fastorder.com
Internal IP: 10.100.1.54 (from topology: db-coordinator-postgresql)
Shared IP: 142.93.238.16 (legacy/public, also included in SANs)
Coordinator variants:
- db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
- db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 06:39:47 UTC] USER=www-data EUID=0 PID=4172039 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator and /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:39:47 UTC] USER=www-data EUID=0 PID=4172048 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-02-05 06:39:47 UTC] USER=www-data EUID=0 PID=4172058 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-4172004
[2026-02-05 06:39:47 UTC] USER=www-data EUID=0 PID=4172067 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-4172004/ra_root.crt
[2026-02-05 06:39:47 UTC] USER=www-data EUID=0 PID=4172076 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-4172004/ra_root.key
[2026-02-05 06:39:47 UTC] USER=www-data EUID=0 PID=4172085 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4172004/ra_root.crt
[2026-02-05 06:39:47 UTC] USER=www-data EUID=0 PID=4172094 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4172004/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[2026-02-05 06:39:48 UTC] USER=www-data EUID=0 PID=4172163 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4172004/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 06:39:48 UTC] USER=www-data EUID=0 PID=4172178 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4172004/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 06:39:48 UTC] USER=www-data EUID=0 PID=4172191 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-02-05 06:39:48 UTC] USER=www-data EUID=0 PID=4172201 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4172004/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:48 UTC] USER=www-data EUID=0 PID=4172215 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:48 UTC] USER=www-data EUID=0 PID=4172227 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:39:48 UTC] USER=www-data EUID=0 PID=4172238 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 06:39:48 UTC] USER=www-data EUID=0 PID=4172249 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 06:39:48 UTC] USER=www-data EUID=0 PID=4172258 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 06:39:48 UTC] USER=www-data EUID=0 PID=4172267 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 06:39:48 UTC] USER=www-data EUID=0 PID=4172277 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 06:39:48 UTC] USER=www-data EUID=0 PID=4172286 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:39:48 UTC] USER=www-data EUID=0 PID=4172295 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🔍 Verifying certificate...
Certificate details:
Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
--
X509v3 Subject Alternative Name:
DNS:db-web-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:db-web-universe-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-web-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-web-universe-main-dev-postgresql.fastorder.com, IP Address:10.100.1.54, IP Address:127.0.0.1, IP Address:142.93.238.16
X509v3 Subject Key Identifier:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Node: coordinator
Primary CN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-coordinator.service
3. Test SSL connection:
psql "host=db-web-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-02-05 06:39:49 UTC] USER=www-data EUID=0 PID=4172328 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.crt
[2026-02-05 06:39:49 UTC] USER=www-data EUID=0 PID=4172337 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/server.key
[2026-02-05 06:39:49 UTC] USER=www-data EUID=0 PID=4172346 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[0;32m[OK][0m mTLS certificates OK (server cert + client certs verified) and keys secured
[0;34m[INFO][0m Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-02-05 06:39:49 UTC] USER=www-data EUID=0 PID=4172367 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 06:39:49 UTC] USER=www-data EUID=0 PID=4172391 ACTION=passthru ARGS=systemctl stop postgresql
[1;33m[WARN][0m Cleaning stale socket directory /var/run/postgresql-web-universe-main-dev-coordinator
[2026-02-05 06:39:49 UTC] USER=www-data EUID=0 PID=4172425 ACTION=fsop ARGS=rm -rf /var/run/postgresql-web-universe-main-dev-coordinator
[0;32m[OK][0m No conflicting Postgres left on port 5432
[0;32m[OK][0m Using postgres password from vault provider
[2026-02-05 06:39:49 UTC] USER=www-data EUID=0 PID=4172488 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.ngIxSl
[2026-02-05 06:39:49 UTC] USER=www-data EUID=0 PID=4172509 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.ngIxSl
[2026-02-05 06:39:49 UTC] USER=www-data EUID=0 PID=4172532 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev
[2026-02-05 06:39:49 UTC] USER=www-data EUID=0 PID=4172556 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev
[2026-02-05 06:39:50 UTC] USER=www-data EUID=0 PID=4172578 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/web-universe-main-dev
[0;34m[INFO][0m Initializing cluster in /data/postgresql/17/web-universe-main-dev/coordinator (SCRAM; pwfile)
[1;33m[WARN][0m Removing existing data directory: /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 06:39:50 UTC] USER=www-data EUID=0 PID=4172599 ACTION=fsop ARGS=rm -rf /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 06:39:50 UTC] USER=www-data EUID=0 PID=4172628 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 06:39:50 UTC] USER=www-data EUID=0 PID=4172649 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 06:39:50 UTC] USER=www-data EUID=0 PID=4172675 ACTION=fsop ARGS=chmod 700 /data/postgresql/17/web-universe-main-dev/coordinator
[2026-02-05 06:39:50 UTC] USER=www-data EUID=0 PID=4172696 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-web-universe-main-dev-coordinator
[2026-02-05 06:39:50 UTC] USER=www-data EUID=0 PID=4172739 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-web-universe-main-dev-coordinator
[2026-02-05 06:39:50 UTC] USER=www-data EUID=0 PID=4172748 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/web-universe-main-dev/coordinator --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.ngIxSl
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /data/postgresql/17/web-universe-main-dev/coordinator ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
/usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/web-universe-main-dev/coordinator -l logfile start
[0;32m[OK][0m initdb complete
[2026-02-05 06:39:51 UTC] USER=www-data EUID=0 PID=4172786 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.ngIxSl
[0;34m[INFO][0m Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[0;32m[OK][0m postgresql.conf updated successfully
[0;34m[INFO][0m Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-02-05 06:39:51 UTC] USER=www-data EUID=0 PID=4172841 ACTION=fsop ARGS=cp /tmp/tmp.76O2v9AHXV /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
[2026-02-05 06:39:51 UTC] USER=www-data EUID=0 PID=4172862 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
[2026-02-05 06:39:51 UTC] USER=www-data EUID=0 PID=4172883 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
[0;32m[OK][0m pg_hba.conf updated
[0;34m[INFO][0m Creating systemd unit: /etc/systemd/system/postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 06:39:51 UTC] USER=www-data EUID=0 PID=4172908 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.FBrTwp /etc/systemd/system/postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 06:39:51 UTC] USER=www-data EUID=0 PID=4172929 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m systemd unit written
[2026-02-05 06:39:52 UTC] USER=www-data EUID=0 PID=4172950 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 06:39:52 UTC] USER=www-data EUID=0 PID=4172972 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 06:39:52 UTC] USER=www-data EUID=0 PID=4172993 ACTION=passthru ARGS=systemctl daemon-reload
[0;34m[INFO][0m Starting PostgreSQL instance...
[2026-02-05 06:39:53 UTC] USER=www-data EUID=0 PID=4173127 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[0;34m[INFO][0m Waiting for ACTIVE (systemd)…
[2026-02-05 06:39:53 UTC] USER=www-data EUID=0 PID=4173221 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m Service ACTIVE
[0;34m[INFO][0m Waiting for port 5432 bind…
[0;32m[OK][0m Port bound
[0;34m[INFO][0m Waiting pg_isready (socket)…
[0;32m[OK][0m Readiness via socket OK
[0;34m[INFO][0m Waiting pg_isready (TCP db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432)…
[0;32m[OK][0m Startup sequence complete
[0;34m[INFO][0m Validating core security GUCs (via local socket)…
[0;32m[OK][0m Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[0;34m[INFO][0m Provisioning application database and Debezium role (if not exists)...
[0;34m[INFO][0m Checking if database fastorder_web_universe_main_dev_db exists...
[0;34m[INFO][0m DB check result: exit_code=0, output='[2026-02-05 06:39:54 UTC] USER=www-data EUID=0 PID=4173394 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_web_universe_main_dev_db''
[0;34m[INFO][0m Creating database fastorder_web_universe_main_dev_db...
[2026-02-05 06:39:54 UTC] USER=www-data EUID=0 PID=4173417 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_web_universe_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[0;32m[OK][0m Database fastorder_web_universe_main_dev_db created
[0;34m[INFO][0m Checking if role debezium_user exists...
[0;34m[INFO][0m Role check result: exit_code=0, output='[2026-02-05 06:39:55 UTC] USER=www-data EUID=0 PID=4173449 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[0;34m[INFO][0m Creating role debezium_user...
[2026-02-05 06:39:55 UTC] USER=www-data EUID=0 PID=4173480 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD '5c6hf8XiznhhzvPpXQe9Q4tr';
CREATE ROLE
[0;32m[OK][0m Role debezium_user created
[2026-02-05 06:39:55 UTC] USER=www-data EUID=0 PID=4173516 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_web_universe_main_dev_db" TO debezium_user;
GRANT
[0;32m[OK][0m Application DB (fastorder_web_universe_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[0;34m[INFO][0m Applying connection and memory optimizations...
[0;34m[INFO][0m Current settings: max_connections=100, work_mem=4MB
[0;34m[INFO][0m Target settings (coordinator): max_connections=150, work_mem=8MB
[2026-02-05 06:39:56 UTC] USER=www-data EUID=0 PID=4173596 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 150;
ALTER SYSTEM
[2026-02-05 06:39:56 UTC] USER=www-data EUID=0 PID=4173630 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-02-05 06:39:56 UTC] USER=www-data EUID=0 PID=4173656 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
pg_reload_conf
----------------
t
(1 row)
[0;32m[OK][0m Settings applied to postgresql.auto.conf
[2026-02-05 06:39:56 UTC] USER=www-data EUID=0 PID=4173673 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/coordinator/standby.signal
[0;34m[INFO][0m Service recently started (3s ago) - restarting to apply max_connections...
[0;34m[INFO][0m Stopping service...
[2026-02-05 06:39:56 UTC] USER=www-data EUID=0 PID=4173696 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[0;34m[INFO][0m Waiting for port 5432 to be released...
[0;32m[OK][0m Port 5432 released
[0;34m[INFO][0m Starting service...
[2026-02-05 06:40:00 UTC] USER=www-data EUID=0 PID=4173924 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 06:40:05 UTC] USER=www-data EUID=0 PID=4174157 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m ✅ Optimization complete: max_connections=150, work_mem=8MB
[0;34m[INFO][0m Setting postgres password via centralized script... for coordinator
[0;34m[INFO][0m Temporarily disabling synchronous_commit on coordinator for password setting...
[0;32m[OK][0m Disabled synchronous_commit (was: on)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: coordinator[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-coordinator[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mInitial setup: Using password from initdb[0m
[0;32m✓ PostgreSQL password already set during initdb[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/coordinator
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/coordinator[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials coordinator[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
[0;34m[INFO][0m Restoring synchronous_commit on coordinator...
[0;32m[OK][0m Restored synchronous_commit to: on
[0;32m[OK][0m Password set and persisted
[0;34m[INFO][0m Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.54
[0;34m[INFO][0m Primary hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[0;34m[INFO][0m Adding /etc/hosts entries for coordinator...
[0;34m[INFO][0m 1. db-web-universe-main-dev-postgresql.fastorder.com → 10.100.1.54 (primary/short)
[0;34m[INFO][0m 2. db-web-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.54 (compatibility)
[0;34m[INFO][0m ✅ db-web-universe-main-dev-postgresql.fastorder.com already exists with correct IP
[0;34m[INFO][0m ✅ db-web-universe-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
10.100.1.54 db-web-universe-main-dev-postgresql.fastorder.com
10.100.1.54 db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[0;32m[OK][0m PostgreSQL 'web-universe-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt \
sslcert=/home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.crt \
sslkey=/home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/postgres.key \
host=db-web-universe-main-dev-postgresql-coordinator port=5432 dbname=postgres user=postgres"
File been compeleted perfectly: 02-setup-pg-instance
[0;34m[INFO][0m Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: PostgreSQL
[INFO] Identifier: web-universe-main-dev-postgresql-coordinator
[INFO] Identifier Parent: coordinator
[INFO] IP: 10.100.1.54
[INFO] Port: 5432
[INFO] FQDN: db-web-universe-main-dev-postgresql-coordinator
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 90feffd7-89fb-4afb-a63f-cc975d7e928c
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[0;32m[OK][0m PostgreSQL node registered to observability API
[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=da95d2d5-7a04-4279-ae1c-c482185a88f1)
[INFO] 📦 03 role...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 06:40:18 UTC] USER=www-data EUID=0 PID=4174962 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/coordinator/standby.signal
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: debezium_user
Identifier: coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: coordinator
User (CN): debezium_user
Hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 06:40:20 UTC] USER=www-data EUID=0 PID=4175175 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-debezium_user
[2026-02-05 06:40:20 UTC] USER=www-data EUID=0 PID=4175193 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-debezium_user/ra_root.key
[2026-02-05 06:40:20 UTC] USER=www-data EUID=0 PID=4175202 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-02-05 06:40:20 UTC] USER=www-data EUID=0 PID=4175211 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:40:20 UTC] USER=www-data EUID=0 PID=4175242 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:40:20 UTC] USER=www-data EUID=0 PID=4175251 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 06:40:20 UTC] USER=www-data EUID=0 PID=4175260 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 06:40:20 UTC] USER=www-data EUID=0 PID=4175270 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:40:20 UTC] USER=www-data EUID=0 PID=4175280 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:40:20 UTC] USER=www-data EUID=0 PID=4175289 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 06:40:20 UTC] USER=www-data EUID=0 PID=4175298 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 06:40:20 UTC] USER=www-data EUID=0 PID=4175307 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175316 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175325 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175334 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175349 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175359 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175368 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175377 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175386 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175439 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175448 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175466 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175475 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175484 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175493 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175503 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175513 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175522 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175546 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175555 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175564 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:40:21 UTC] USER=www-data EUID=0 PID=4175574 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175583 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175592 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175603 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175615 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175624 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175633 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175642 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175651 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175661 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175671 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175680 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175698 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175717 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175735 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175744 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175753 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175762 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175771 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175781 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175791 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:22 UTC] USER=www-data EUID=0 PID=4175800 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175811 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175820 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175833 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175842 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175851 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175860 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175869 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175878 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175887 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_der.key
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175896 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user_pk8.der
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175906 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175916 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175925 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175934 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175943 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175952 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175961 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175970 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175979 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:40:23 UTC] USER=www-data EUID=0 PID=4175989 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: debezium_user
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/debezium_user.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U debezium_user -d postgres
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: coordinator[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-coordinator[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mGenerating new secure password...[0m
[0;34mUser fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: fastorder_admin_gd[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials coordinator[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: coordinator/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-web-universe-main-dev-postgresql.fastorder.com
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: fastorder_admin_gd
Identifier: coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: coordinator
User (CN): fastorder_admin_gd
Hostname: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 06:40:33 UTC] USER=www-data EUID=0 PID=4176695 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-fastorder_admin_gd
[2026-02-05 06:40:33 UTC] USER=www-data EUID=0 PID=4176704 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-02-05 06:40:33 UTC] USER=www-data EUID=0 PID=4176725 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-02-05 06:40:33 UTC] USER=www-data EUID=0 PID=4176734 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:40:33 UTC] USER=www-data EUID=0 PID=4176752 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:40:33 UTC] USER=www-data EUID=0 PID=4176761 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:40:33 UTC] USER=www-data EUID=0 PID=4176770 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 06:40:33 UTC] USER=www-data EUID=0 PID=4176779 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 06:40:33 UTC] USER=www-data EUID=0 PID=4176788 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:40:33 UTC] USER=www-data EUID=0 PID=4176798 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:40:33 UTC] USER=www-data EUID=0 PID=4176807 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 06:40:33 UTC] USER=www-data EUID=0 PID=4176816 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4176834 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4176843 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4176852 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4176861 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4176885 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4176896 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4176905 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4176914 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4176923 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4176932 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4176941 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4176950 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4176976 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4176985 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4176997 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4177014 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4177034 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4177052 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4177064 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4177073 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4177082 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4177091 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4177100 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4177109 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4177119 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4177129 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4177139 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4177149 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4177158 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:34 UTC] USER=www-data EUID=0 PID=4177177 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177186 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177195 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177204 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177213 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177222 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177232 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177242 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177252 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177261 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177270 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177279 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177288 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177297 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177306 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177315 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177324 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177333 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177342 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177351 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177361 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177371 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177380 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177389 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177398 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177407 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177416 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177425 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177434 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177443 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177452 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177461 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177472 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177484 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177496 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177505 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177515 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 06:40:35 UTC] USER=www-data EUID=0 PID=4177525 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 06:40:36 UTC] USER=www-data EUID=0 PID=4177534 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 06:40:36 UTC] USER=www-data EUID=0 PID=4177543 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:40:36 UTC] USER=www-data EUID=0 PID=4177552 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:40:36 UTC] USER=www-data EUID=0 PID=4177561 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:40:36 UTC] USER=www-data EUID=0 PID=4177570 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: fastorder_admin_gd
Node: coordinator
FQDN: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/fastorder_admin_gd.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-coordinator.fastorder.com -U fastorder_admin_gd -d postgres
🧱 Connecting via Unix socket to create role and database...
Socket: /var/run/postgresql-web-universe-main-dev-coordinator:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️ Database fastorder_web_universe_main_dev_db already exists, skipping creation
[2026-02-05 06:40:36 UTC] USER=www-data EUID=0 PID=4177630 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️ Using pg_hba.conf: /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-02-05 06:40:36 UTC] USER=www-data EUID=0 PID=4177673 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[0;34m=== Pre-flight Checks ===[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;32m✓[0m AWS Secrets Manager accessible
[0;34m=== Retrieving Credentials from AWS ===[0m
ℹ️ Retrieving PostgreSQL credentials for: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️ Fetching secret: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials loaded for coordinator/fastorder_admin_gd: fastorder_admin_gd@db-web-universe-main-dev-postgresql.fastorder.com:5432/fastorder_web_universe_main_dev_db
[0;32m✓[0m Credentials retrieved: fastorder_admin_gd@db-web-universe-main-dev-postgresql.fastorder.com:5432/fastorder_web_universe_main_dev_db
[0;34m╔════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Test Suite (AWS Secrets MGR) ║[0m
[0;34m╚════════════════════════════════════════════╝[0m
[0;34m=== PostgreSQL Authentication Test ===[0m
[0;31m✗[0m PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-web-universe-main-dev-postgresql.fastorder.com" (10.100.1.54), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/web/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 06:40:41 UTC] USER=www-data EUID=0 PID=4177994 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/coordinator/standby.signal
── fast setup ─────────────────────────────────────────────
NAME : web-universe-main-dev
IDENTIFIER : coordinator
PG HOST : db-web-universe-main-dev-postgresql.fastorder.com:5432
ROLE : debezium_user
DB : fastorder_web_universe_main_dev_db
SCHEMA : web
AUTH MODE : scram (scram=password over TLS | cert=mTLS)
SUBNET ALLOW: 10.201.0.0/16
CONNECT /32 : 142.93.238.16
SSL DIR : /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator
DNS → 10.100.1.54
CA : /home/www-data/ssl/.postgresql/web-universe-main-dev/coordinator/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: coordinator[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-coordinator[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mGenerating new secure password...[0m
[0;34mUser debezium_user does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: debezium_user[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/coordinator/debezium_user[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials coordinator[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: coordinator/debezium_user
✓ Retrieved password from secrets vault
password : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️ Role debezium_user exists, updating
[2026-02-05 06:40:48 UTC] USER=www-data EUID=0 PID=4178435 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️ Database fastorder_web_universe_main_dev_db already exists
[2026-02-05 06:40:48 UTC] USER=www-data EUID=0 PID=4178470 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-coordinator -p 5432 -d fastorder_web_universe_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️ Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/web-universe-main-dev/coordinator/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=da95d2d5-7a04-4279-ae1c-c482185a88f1)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
ℹ️ Service-specific setup (web) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)
🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=web
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
🔍 DEBUG_CHECKPOINT_03: Found service-specific run.sh for: web
[DEBUG] Tracking substep start: steps/01-install/steps/web (RUN_UUID=da95d2d5-7a04-4279-ae1c-c482185a88f1)
[INFO] 🔸 Service: web (service-specific)
🔍 DEBUG_CHECKPOINT_04: Executing service-specific: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Web Service Schema Provisioning
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service: web
[INFO] Zone: universe
[INFO] Branch: main
[INFO] Environment: dev
[INFO] Identifier: coordinator
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] 📦 Running: 01-init-schema.sh...
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Initializing config schema
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Identifier: coordinator
Database: fastorder_web_universe_main_dev_db
Host: db-web-universe-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking if config schema exists...
[INFO] Creating config schema and tables...
CREATE EXTENSION
CREATE SCHEMA
CREATE TABLE
COMMENT
CREATE TABLE
COMMENT
CREATE TABLE
COMMENT
CREATE INDEX
CREATE INDEX
CREATE INDEX
INSERT 0 1
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
NOTICE: trigger "trg_public_defaults_version" for relation "config.public_defaults" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
NOTICE: trigger "trg_feature_flags_version" for relation "config.feature_flags" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
NOTICE: trigger "trg_public_defaults_set_updated_at" for relation "config.public_defaults" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
NOTICE: trigger "trg_feature_flags_set_updated_at" for relation "config.feature_flags" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
[OK] Config schema and tables created
[INFO] Seeding initial guest services data...
INSERT 0 9
INSERT 0 1
[OK] Initial data seeded
[INFO] Verifying config schema...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Config Schema Verification
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Guest services: 9
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] Config schema initialization complete
[OK] ✓ 01-init-schema.sh completed
[INFO] ═══════════════════════════════════════════════════════════════
[OK] Web service schema provisioning complete
[INFO] ═══════════════════════════════════════════════════════════════
[0;32m✓[0m ✅ Coordinator setup completed
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up 1 worker(s) (Citus data nodes)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up worker: worker-01
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-02-05 06:40:53 UTC] USER=unknown EUID=33 PID=4178778 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 06:40:53 UTC] USER=unknown EUID=33 PID=4178785 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 06:40:53 UTC] USER=unknown EUID=33 PID=4178792 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-02-05 06:40:53 UTC] USER=unknown EUID=33 PID=4178799 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 06:40:53 UTC] USER=unknown EUID=33 PID=4178806 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-02-05 06:40:53 UTC] USER=unknown EUID=33 PID=4178813 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for web in universe-dev...
[INFO] Environment: web-universe-main-dev
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=da95d2d5-7a04-4279-ae1c-c482185a88f1 JOB_UUID=32abb742-02ad-4ac9-8236-502afdefc367
[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=da95d2d5-7a04-4279-ae1c-c482185a88f1)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.243
[0;34m[INFO][0m Primary hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[0;34m[INFO][0m Adding /etc/hosts entry for worker-01...
[0;34m[INFO][0m db-web-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.243
[0;34m[INFO][0m ✅ db-web-universe-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
10.100.1.243 db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=da95d2d5-7a04-4279-ae1c-c482185a88f1)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe (FastOrder Universe)
Branch: main
Env: dev
Node: worker-01
Primary CN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Alt CN: web-universe-main-dev.fastorder.com
Internal IP: 10.100.1.243 (from topology: db-worker-01-postgresql)
Shared IP: 142.93.238.16 (legacy/public, also included in SANs)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01 and /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-02-05 06:40:57 UTC] USER=www-data EUID=0 PID=4179076 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-4179000
[2026-02-05 06:40:57 UTC] USER=www-data EUID=0 PID=4179086 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-4179000/ra_root.crt
[2026-02-05 06:40:57 UTC] USER=www-data EUID=0 PID=4179096 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-4179000/ra_root.key
[2026-02-05 06:40:57 UTC] USER=www-data EUID=0 PID=4179105 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4179000/ra_root.crt
[2026-02-05 06:40:57 UTC] USER=www-data EUID=0 PID=4179114 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4179000/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[2026-02-05 06:40:59 UTC] USER=www-data EUID=0 PID=4179172 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4179000/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 06:40:59 UTC] USER=www-data EUID=0 PID=4179188 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4179000/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 06:40:59 UTC] USER=www-data EUID=0 PID=4179202 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-02-05 06:40:59 UTC] USER=www-data EUID=0 PID=4179222 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:40:59 UTC] USER=www-data EUID=0 PID=4179231 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:40:59 UTC] USER=www-data EUID=0 PID=4179240 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 06:40:59 UTC] USER=www-data EUID=0 PID=4179251 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 06:41:00 UTC] USER=www-data EUID=0 PID=4179269 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 06:41:00 UTC] USER=www-data EUID=0 PID=4179278 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 06:41:00 UTC] USER=www-data EUID=0 PID=4179288 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:00 UTC] USER=www-data EUID=0 PID=4179301 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔍 Verifying certificate...
Certificate details:
Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
--
X509v3 Subject Alternative Name:
DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01, DNS:localhost, IP Address:10.100.1.243, IP Address:127.0.0.1, IP Address:142.93.238.16
X509v3 Subject Key Identifier:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Node: worker-01
Primary CN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-worker-01.service
3. Test SSL connection:
psql "host=db-web-universe-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: postgres
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: worker-01
User (CN): postgres
Hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 06:41:00 UTC] USER=www-data EUID=0 PID=4179390 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-02-05 06:41:00 UTC] USER=www-data EUID=0 PID=4179408 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 06:41:00 UTC] USER=www-data EUID=0 PID=4179417 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-02-05 06:41:00 UTC] USER=www-data EUID=0 PID=4179436 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:01 UTC] USER=www-data EUID=0 PID=4179467 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:01 UTC] USER=www-data EUID=0 PID=4179488 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:01 UTC] USER=www-data EUID=0 PID=4179507 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 06:41:01 UTC] USER=www-data EUID=0 PID=4179516 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:01 UTC] USER=www-data EUID=0 PID=4179539 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 06:41:01 UTC] USER=www-data EUID=0 PID=4179557 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 06:41:01 UTC] USER=www-data EUID=0 PID=4179566 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 06:41:01 UTC] USER=www-data EUID=0 PID=4179580 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:01 UTC] USER=www-data EUID=0 PID=4179590 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:01 UTC] USER=www-data EUID=0 PID=4179604 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 06:41:01 UTC] USER=www-data EUID=0 PID=4179615 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 06:41:01 UTC] USER=www-data EUID=0 PID=4179624 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179633 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179642 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179679 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179688 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179697 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179707 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179716 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179725 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179734 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179743 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179752 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179761 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179790 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179800 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179809 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179818 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179836 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179846 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179858 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 06:41:02 UTC] USER=www-data EUID=0 PID=4179867 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4179876 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4179885 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4179894 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4179904 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4179922 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4179932 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4179941 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4179959 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4179968 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4179977 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4179986 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4179995 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4180004 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4180014 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4180026 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4180035 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4180045 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4180064 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:03 UTC] USER=www-data EUID=0 PID=4180073 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180082 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180091 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180101 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180110 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180119 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180128 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180146 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180155 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180165 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180175 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180186 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180199 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180208 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180219 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180231 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180240 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180249 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:41:04 UTC] USER=www-data EUID=0 PID=4180258 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: postgres
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: postgres
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: worker-01
User (CN): postgres
Hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 06:41:05 UTC] USER=www-data EUID=0 PID=4180317 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-02-05 06:41:05 UTC] USER=www-data EUID=0 PID=4180326 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 06:41:05 UTC] USER=www-data EUID=0 PID=4180335 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-02-05 06:41:05 UTC] USER=www-data EUID=0 PID=4180344 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-02-05 06:41:05 UTC] USER=www-data EUID=0 PID=4180353 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:05 UTC] USER=www-data EUID=0 PID=4180368 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:05 UTC] USER=www-data EUID=0 PID=4180377 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:05 UTC] USER=www-data EUID=0 PID=4180386 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 06:41:05 UTC] USER=www-data EUID=0 PID=4180395 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 06:41:05 UTC] USER=www-data EUID=0 PID=4180404 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:05 UTC] USER=www-data EUID=0 PID=4180413 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:05 UTC] USER=www-data EUID=0 PID=4180422 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 06:41:05 UTC] USER=www-data EUID=0 PID=4180441 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 06:41:05 UTC] USER=www-data EUID=0 PID=4180460 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 06:41:05 UTC] USER=www-data EUID=0 PID=4180470 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 06:41:05 UTC] USER=www-data EUID=0 PID=4180481 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 06:41:05 UTC] USER=www-data EUID=0 PID=4180490 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180499 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180508 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180517 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180526 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180536 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180546 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180557 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180566 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180592 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180619 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180628 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180646 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180656 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180666 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180675 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180695 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180715 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180724 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180740 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180754 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180763 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:06 UTC] USER=www-data EUID=0 PID=4180772 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180781 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180790 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180799 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180808 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180818 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180827 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180837 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180847 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180865 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180874 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180883 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180892 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180902 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180911 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180920 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180929 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180938 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180949 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180961 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180973 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180982 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4180991 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4181002 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4181011 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4181020 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4181029 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4181038 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4181047 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4181056 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4181065 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_der.key
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4181074 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/postgres_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres_pk8.der
[2026-02-05 06:41:07 UTC] USER=www-data EUID=0 PID=4181084 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:41:08 UTC] USER=www-data EUID=0 PID=4181094 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:41:08 UTC] USER=www-data EUID=0 PID=4181103 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:41:08 UTC] USER=www-data EUID=0 PID=4181112 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 06:41:08 UTC] USER=www-data EUID=0 PID=4181122 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 06:41:08 UTC] USER=www-data EUID=0 PID=4181131 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 06:41:08 UTC] USER=www-data EUID=0 PID=4181140 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:41:08 UTC] USER=www-data EUID=0 PID=4181149 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:41:08 UTC] USER=www-data EUID=0 PID=4181158 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: postgres
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres
[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=da95d2d5-7a04-4279-ae1c-c482185a88f1)
[INFO] 📦 02 setup pg instance...
[0;34m[DEADLOCK-PREVENTION][0m Deadlock prevention library loaded
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Using existing db-worker-01-postgresql environment: db-web-universe-main-dev-postgresql-worker-01.fastorder.com (10.100.1.243)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.243
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m Data dir: /data/postgresql/17/web-universe-main-dev/worker-01
[0;34m[INFO][0m Port: 5432
[0;34m[INFO][0m Hostname: db-web-universe-main-dev-postgresql-worker-01
[2026-02-05 06:41:10 UTC] USER=www-data EUID=0 PID=4181329 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:10 UTC] USER=www-data EUID=0 PID=4181361 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:10 UTC] USER=www-data EUID=0 PID=4181385 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[1;33m[WARN][0m Server certificate not found at /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[0;34m[INFO][0m Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe (FastOrder Universe)
Branch: main
Env: dev
Node: worker-01
Primary CN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Alt CN: web-universe-main-dev.fastorder.com
Internal IP: 10.100.1.243 (from topology: db-worker-01-postgresql)
Shared IP: 142.93.238.16 (legacy/public, also included in SANs)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-02-05 06:41:11 UTC] USER=www-data EUID=0 PID=4181470 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01 and /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-02-05 06:41:11 UTC] USER=www-data EUID=0 PID=4181494 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-4181393
[2026-02-05 06:41:11 UTC] USER=www-data EUID=0 PID=4181503 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-4181393/ra_root.crt
[2026-02-05 06:41:11 UTC] USER=www-data EUID=0 PID=4181518 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-4181393/ra_root.key
[2026-02-05 06:41:11 UTC] USER=www-data EUID=0 PID=4181528 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4181393/ra_root.crt
[2026-02-05 06:41:11 UTC] USER=www-data EUID=0 PID=4181537 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4181393/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[2026-02-05 06:41:14 UTC] USER=www-data EUID=0 PID=4181637 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4181393/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 06:41:14 UTC] USER=www-data EUID=0 PID=4181657 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4181393/server.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 06:41:14 UTC] USER=www-data EUID=0 PID=4181673 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-02-05 06:41:14 UTC] USER=www-data EUID=0 PID=4181686 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4181393/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:14 UTC] USER=www-data EUID=0 PID=4181695 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:14 UTC] USER=www-data EUID=0 PID=4181704 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:15 UTC] USER=www-data EUID=0 PID=4181713 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-02-05 06:41:15 UTC] USER=www-data EUID=0 PID=4181725 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 06:41:15 UTC] USER=www-data EUID=0 PID=4181754 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 06:41:15 UTC] USER=www-data EUID=0 PID=4181777 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
[2026-02-05 06:41:15 UTC] USER=www-data EUID=0 PID=4181786 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:15 UTC] USER=www-data EUID=0 PID=4181795 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🔍 Verifying certificate...
Certificate details:
Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
--
X509v3 Subject Alternative Name:
DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:web-universe-main-dev.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:db-web-universe-main-dev-postgresql-worker-01, DNS:localhost, IP Address:10.100.1.243, IP Address:127.0.0.1, IP Address:142.93.238.16
X509v3 Subject Key Identifier:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Node: worker-01
Primary CN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@web-universe-main-dev-worker-01.service
3. Test SSL connection:
psql "host=db-web-universe-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-02-05 06:41:15 UTC] USER=www-data EUID=0 PID=4181837 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/server.key
[2026-02-05 06:41:15 UTC] USER=www-data EUID=0 PID=4181846 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[0;32m[OK][0m mTLS certificates OK (server cert + client certs verified) and keys secured
[0;34m[INFO][0m Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-02-05 06:41:15 UTC] USER=www-data EUID=0 PID=4181867 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 06:41:16 UTC] USER=www-data EUID=0 PID=4181902 ACTION=passthru ARGS=systemctl stop postgresql
[1;33m[WARN][0m Cleaning stale socket directory /var/run/postgresql-web-universe-main-dev-worker-01
[2026-02-05 06:41:16 UTC] USER=www-data EUID=0 PID=4181939 ACTION=fsop ARGS=rm -rf /var/run/postgresql-web-universe-main-dev-worker-01
[0;32m[OK][0m No conflicting Postgres left on port 5432
[0;32m[OK][0m Using postgres password from vault provider
[2026-02-05 06:41:18 UTC] USER=www-data EUID=0 PID=4182056 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.IrdrYl
[2026-02-05 06:41:18 UTC] USER=www-data EUID=0 PID=4182080 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.IrdrYl
[2026-02-05 06:41:18 UTC] USER=www-data EUID=0 PID=4182107 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev
[2026-02-05 06:41:18 UTC] USER=www-data EUID=0 PID=4182154 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev
[2026-02-05 06:41:19 UTC] USER=www-data EUID=0 PID=4182178 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/web-universe-main-dev
[0;34m[INFO][0m Initializing cluster in /data/postgresql/17/web-universe-main-dev/worker-01 (SCRAM; pwfile)
[1;33m[WARN][0m Removing existing data directory: /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 06:41:19 UTC] USER=www-data EUID=0 PID=4182199 ACTION=fsop ARGS=rm -rf /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 06:41:19 UTC] USER=www-data EUID=0 PID=4182221 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 06:41:19 UTC] USER=www-data EUID=0 PID=4182242 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/worker-01
[2026-02-05 06:41:19 UTC] USER=www-data EUID=0 PID=4182284 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-web-universe-main-dev-worker-01
[2026-02-05 06:41:19 UTC] USER=www-data EUID=0 PID=4182309 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-web-universe-main-dev-worker-01
[2026-02-05 06:41:19 UTC] USER=www-data EUID=0 PID=4182330 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-web-universe-main-dev-worker-01
[2026-02-05 06:41:19 UTC] USER=www-data EUID=0 PID=4182340 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/web-universe-main-dev/worker-01 --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.IrdrYl
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /data/postgresql/17/web-universe-main-dev/worker-01 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
/usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/web-universe-main-dev/worker-01 -l logfile start
[0;32m[OK][0m initdb complete
[2026-02-05 06:41:20 UTC] USER=www-data EUID=0 PID=4182385 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.IrdrYl
[0;34m[INFO][0m Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[0;32m[OK][0m postgresql.conf updated successfully
[0;34m[INFO][0m Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-02-05 06:41:20 UTC] USER=www-data EUID=0 PID=4182445 ACTION=fsop ARGS=cp /tmp/tmp.VXd2pM2FSP /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
[2026-02-05 06:41:21 UTC] USER=www-data EUID=0 PID=4182490 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
[2026-02-05 06:41:21 UTC] USER=www-data EUID=0 PID=4182513 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
[0;32m[OK][0m pg_hba.conf updated
[0;34m[INFO][0m Creating systemd unit: /etc/systemd/system/postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 06:41:21 UTC] USER=www-data EUID=0 PID=4182544 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.gVKelR /etc/systemd/system/postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 06:41:21 UTC] USER=www-data EUID=0 PID=4182565 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@web-universe-main-dev-worker-01.service
[0;32m[OK][0m systemd unit written
[2026-02-05 06:41:21 UTC] USER=www-data EUID=0 PID=4182617 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-02-05 06:41:21 UTC] USER=www-data EUID=0 PID=4182638 ACTION=passthru ARGS=systemctl daemon-reload
[0;34m[INFO][0m Starting PostgreSQL instance...
[2026-02-05 06:41:23 UTC] USER=www-data EUID=0 PID=4182764 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-worker-01.service
[0;34m[INFO][0m Waiting for ACTIVE (systemd)…
[2026-02-05 06:41:23 UTC] USER=www-data EUID=0 PID=4182822 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-worker-01.service
[0;32m[OK][0m Service ACTIVE
[0;34m[INFO][0m Waiting for port 5432 bind…
[0;32m[OK][0m Port bound
[0;34m[INFO][0m Waiting pg_isready (socket)…
[0;32m[OK][0m Readiness via socket OK
[0;34m[INFO][0m Waiting pg_isready (TCP db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432)…
[0;32m[OK][0m Startup sequence complete
[0;34m[INFO][0m Validating core security GUCs (via local socket)…
[0;32m[OK][0m Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[0;34m[INFO][0m Provisioning application database and Debezium role (if not exists)...
[0;34m[INFO][0m Checking if database fastorder_web_universe_main_dev_db exists...
[0;34m[INFO][0m DB check result: exit_code=0, output='[2026-02-05 06:41:24 UTC] USER=www-data EUID=0 PID=4183004 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_web_universe_main_dev_db''
[0;34m[INFO][0m Creating database fastorder_web_universe_main_dev_db...
[2026-02-05 06:41:24 UTC] USER=www-data EUID=0 PID=4183027 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_web_universe_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[0;32m[OK][0m Database fastorder_web_universe_main_dev_db created
[0;34m[INFO][0m Checking if role debezium_user exists...
[0;34m[INFO][0m Role check result: exit_code=0, output='[2026-02-05 06:41:25 UTC] USER=www-data EUID=0 PID=4183054 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[0;34m[INFO][0m Creating role debezium_user...
[2026-02-05 06:41:25 UTC] USER=www-data EUID=0 PID=4183099 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'vkP76iCsRW6kXFvjGZatm2tn';
CREATE ROLE
[0;32m[OK][0m Role debezium_user created
[2026-02-05 06:41:25 UTC] USER=www-data EUID=0 PID=4183132 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_web_universe_main_dev_db" TO debezium_user;
GRANT
[0;32m[OK][0m Application DB (fastorder_web_universe_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[0;34m[INFO][0m Applying connection and memory optimizations...
[0;34m[INFO][0m Current settings: max_connections=100, work_mem=4MB
[0;34m[INFO][0m Target settings (worker): max_connections=100, work_mem=8MB
[2026-02-05 06:41:25 UTC] USER=www-data EUID=0 PID=4183212 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 100;
ALTER SYSTEM
[2026-02-05 06:41:26 UTC] USER=www-data EUID=0 PID=4183235 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-02-05 06:41:26 UTC] USER=www-data EUID=0 PID=4183270 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
pg_reload_conf
----------------
t
(1 row)
[0;32m[OK][0m Settings applied to postgresql.auto.conf
[2026-02-05 06:41:26 UTC] USER=www-data EUID=0 PID=4183290 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/worker-01/standby.signal
[0;34m[INFO][0m Service recently started (3s ago) - restarting to apply max_connections...
[0;34m[INFO][0m Stopping service...
[2026-02-05 06:41:26 UTC] USER=www-data EUID=0 PID=4183312 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-worker-01.service
[0;34m[INFO][0m Waiting for port 5432 to be released...
[0;32m[OK][0m Port 5432 released
[0;34m[INFO][0m Starting service...
[2026-02-05 06:41:29 UTC] USER=www-data EUID=0 PID=4183441 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-worker-01.service
[2026-02-05 06:41:35 UTC] USER=www-data EUID=0 PID=4183724 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-worker-01.service
[0;32m[OK][0m ✅ Optimization complete: max_connections=100, work_mem=8MB
[0;32m[OK][0m Synchronous replication already configured (synchronous_commit: on)
[0;34m[INFO][0m Setting postgres password via centralized script... for worker-01
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: worker-01[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-worker-01[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mInitial setup: Using password from initdb[0m
[0;32m✓ PostgreSQL password already set during initdb[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/worker-01
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/worker-01[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials worker-01[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
[0;32m[OK][0m Password set and persisted
[0;34m[INFO][0m Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.243
[0;34m[INFO][0m Primary hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[0;34m[INFO][0m Adding /etc/hosts entry for worker-01...
[0;34m[INFO][0m db-web-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.243
[0;34m[INFO][0m ✅ db-web-universe-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
10.100.1.243 db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[0;32m[OK][0m PostgreSQL 'web-universe-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt \
sslcert=/home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.crt \
sslkey=/home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/postgres.key \
host=db-web-universe-main-dev-postgresql-worker-01 port=5432 dbname=postgres user=postgres"
File been compeleted perfectly: 02-setup-pg-instance
[0;34m[INFO][0m Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: PostgreSQL
[INFO] Identifier: web-universe-main-dev-postgresql-worker-01
[INFO] Identifier Parent: worker-01
[INFO] IP: 10.100.1.243
[INFO] Port: 5432
[INFO] FQDN: db-web-universe-main-dev-postgresql-worker-01
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 84e4185a-2ef1-49c1-8d2a-841d077f036b
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[0;32m[OK][0m PostgreSQL node registered to observability API
[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=da95d2d5-7a04-4279-ae1c-c482185a88f1)
[INFO] 📦 03 role...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 06:41:45 UTC] USER=www-data EUID=0 PID=4184449 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/worker-01/standby.signal
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: debezium_user
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: worker-01
User (CN): debezium_user
Hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 06:41:46 UTC] USER=www-data EUID=0 PID=4184650 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-02-05 06:41:46 UTC] USER=www-data EUID=0 PID=4184661 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-debezium_user/ra_root.key
[2026-02-05 06:41:46 UTC] USER=www-data EUID=0 PID=4184671 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-02-05 06:41:46 UTC] USER=www-data EUID=0 PID=4184682 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184716 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184725 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184735 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184744 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184753 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184762 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184771 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184784 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184802 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184820 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184829 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184838 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184856 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184891 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184900 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184909 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184918 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:47 UTC] USER=www-data EUID=0 PID=4184928 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4184937 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4184946 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4184955 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4184964 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4184973 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4184989 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4184999 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185009 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185018 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185027 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185036 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185045 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185054 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185063 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185072 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185090 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185099 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185121 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185135 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185146 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185155 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185164 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185173 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185182 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 06:41:48 UTC] USER=www-data EUID=0 PID=4185191 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185200 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185209 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185218 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185227 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185236 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185246 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185256 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185265 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185274 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185283 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185292 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185301 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185310 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185319 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185328 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185339 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185348 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_der.key
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185359 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/debezium_user_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user_pk8.der
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185369 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185379 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185397 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185406 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185415 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185424 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185442 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:41:49 UTC] USER=www-data EUID=0 PID=4185451 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: debezium_user
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/debezium_user.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U debezium_user -d postgres
🔐 Generating replicator client certificate for worker-01...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: replicator
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: worker-01
User (CN): replicator
Hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 06:41:50 UTC] USER=www-data EUID=0 PID=4185505 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-02-05 06:41:50 UTC] USER=www-data EUID=0 PID=4185514 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-02-05 06:41:50 UTC] USER=www-data EUID=0 PID=4185532 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185548 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185557 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185567 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185576 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185585 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185594 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185603 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185614 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185634 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185643 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185652 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185661 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185670 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185679 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185688 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185698 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185708 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185751 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185795 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:51 UTC] USER=www-data EUID=0 PID=4185821 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4185830 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4185839 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4185848 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4185857 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4185866 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4185876 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4185885 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4185894 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4185903 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4185921 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4185932 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4185942 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4185951 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4185960 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4185969 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4185988 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4185997 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4186015 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4186033 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4186042 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4186052 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4186062 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:52 UTC] USER=www-data EUID=0 PID=4186071 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186081 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186092 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186101 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186110 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186124 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186133 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186142 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186151 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186160 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186169 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186179 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186198 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186207 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186216 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186225 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186234 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186243 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186252 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186261 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186270 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186280 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_der.key
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186289 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/replicator_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator_pk8.der
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186299 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186309 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186318 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
✅ Symlinked ca.pem
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186337 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 06:41:53 UTC] USER=www-data EUID=0 PID=4186346 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 06:41:54 UTC] USER=www-data EUID=0 PID=4186355 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:41:54 UTC] USER=www-data EUID=0 PID=4186364 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:41:54 UTC] USER=www-data EUID=0 PID=4186375 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:41:54 UTC] USER=www-data EUID=0 PID=4186384 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: replicator
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/replicator.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres
✅ Replicator certificate generated for worker-01
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: worker-01[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-worker-01[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mGenerating new secure password...[0m
[0;34mUser fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: fastorder_admin_gd[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials worker-01[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: worker-01/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: fastorder_admin_gd
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: worker-01
User (CN): fastorder_admin_gd
Hostname: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 06:42:04 UTC] USER=www-data EUID=0 PID=4186970 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-fastorder_admin_gd
[2026-02-05 06:42:04 UTC] USER=www-data EUID=0 PID=4186979 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-02-05 06:42:04 UTC] USER=www-data EUID=0 PID=4186988 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
[2026-02-05 06:42:04 UTC] USER=www-data EUID=0 PID=4186998 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-02-05 06:42:04 UTC] USER=www-data EUID=0 PID=4187007 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:42:04 UTC] USER=www-data EUID=0 PID=4187053 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:42:04 UTC] USER=www-data EUID=0 PID=4187062 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:42:04 UTC] USER=www-data EUID=0 PID=4187071 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 06:42:04 UTC] USER=www-data EUID=0 PID=4187091 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187100 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187109 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187118 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187127 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187136 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187145 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187154 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187163 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187181 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187190 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187199 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187208 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187217 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187226 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187235 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187261 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187270 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187279 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187288 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187297 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187306 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187315 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187325 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187354 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187363 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 06:42:05 UTC] USER=www-data EUID=0 PID=4187373 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:42:06 UTC] USER=www-data EUID=0 PID=4187383 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:42:06 UTC] USER=www-data EUID=0 PID=4187392 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:42:06 UTC] USER=www-data EUID=0 PID=4187415 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:42:06 UTC] USER=www-data EUID=0 PID=4187425 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:42:06 UTC] USER=www-data EUID=0 PID=4187435 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:42:06 UTC] USER=www-data EUID=0 PID=4187444 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 06:42:06 UTC] USER=www-data EUID=0 PID=4187480 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 06:42:06 UTC] USER=www-data EUID=0 PID=4187489 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 06:42:06 UTC] USER=www-data EUID=0 PID=4187498 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 06:42:06 UTC] USER=www-data EUID=0 PID=4187508 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:42:06 UTC] USER=www-data EUID=0 PID=4187527 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:42:06 UTC] USER=www-data EUID=0 PID=4187537 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:42:06 UTC] USER=www-data EUID=0 PID=4187548 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:42:06 UTC] USER=www-data EUID=0 PID=4187557 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:42:06 UTC] USER=www-data EUID=0 PID=4187566 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 06:42:06 UTC] USER=www-data EUID=0 PID=4187575 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 06:42:06 UTC] USER=www-data EUID=0 PID=4187584 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187593 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187602 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187611 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187620 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187630 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187651 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187662 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187671 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187682 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187692 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187703 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187712 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187721 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187732 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187741 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187750 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187759 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187769 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187782 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187793 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187809 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187818 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187827 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187836 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187845 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187854 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:42:07 UTC] USER=www-data EUID=0 PID=4187863 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: fastorder_admin_gd
Node: worker-01
FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/fastorder_admin_gd.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-worker-01.fastorder.com -U fastorder_admin_gd -d postgres
🧱 Connecting via Unix socket to create role and database...
Socket: /var/run/postgresql-web-universe-main-dev-worker-01:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️ Database fastorder_web_universe_main_dev_db already exists, skipping creation
[2026-02-05 06:42:08 UTC] USER=www-data EUID=0 PID=4187923 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️ Using pg_hba.conf: /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-02-05 06:42:08 UTC] USER=www-data EUID=0 PID=4187960 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-worker-01.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[0;34m=== Pre-flight Checks ===[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;32m✓[0m AWS Secrets Manager accessible
[0;34m=== Retrieving Credentials from AWS ===[0m
ℹ️ Retrieving PostgreSQL credentials for: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️ Fetching secret: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials loaded for worker-01/fastorder_admin_gd: fastorder_admin_gd@db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_web_universe_main_dev_db
[0;32m✓[0m Credentials retrieved: fastorder_admin_gd@db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_web_universe_main_dev_db
[0;34m╔════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Test Suite (AWS Secrets MGR) ║[0m
[0;34m╚════════════════════════════════════════════╝[0m
[0;34m=== PostgreSQL Authentication Test ===[0m
[0;31m✗[0m PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-web-universe-main-dev-postgresql-worker-01.fastorder.com" (10.100.1.243), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/web-universe-main-dev/worker-01/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/web/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 06:42:13 UTC] USER=www-data EUID=0 PID=4188290 ACTION=fsop ARGS=test -f /data/postgresql/17/web-universe-main-dev/worker-01/standby.signal
── fast setup ─────────────────────────────────────────────
NAME : web-universe-main-dev
IDENTIFIER : worker-01
PG HOST : db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432
ROLE : debezium_user
DB : fastorder_web_universe_main_dev_db
SCHEMA : web
AUTH MODE : scram (scram=password over TLS | cert=mTLS)
SUBNET ALLOW: 10.201.0.0/16
CONNECT /32 : 142.93.238.16
SSL DIR : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
DNS → 10.100.1.243
CA : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: web[0m
[0;34m Zone: universe[0m
[0;34m Environment: dev[0m
[0;34m Identifier: worker-01[0m
[0;34mAWS Secret: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-web-universe-main-dev-worker-01[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mGenerating new secure password...[0m
[0;34mUser debezium_user does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: debezium_user[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
✅ Secret updated: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/web/universe/main/dev/postgresql/worker-01/debezium_user[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials worker-01[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: worker-01/debezium_user
✓ Retrieved password from secrets vault
password : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️ Role debezium_user exists, updating
[2026-02-05 06:42:20 UTC] USER=www-data EUID=0 PID=4188746 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️ Database fastorder_web_universe_main_dev_db already exists
[2026-02-05 06:42:20 UTC] USER=www-data EUID=0 PID=4188772 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-web-universe-main-dev-worker-01 -p 5432 -d fastorder_web_universe_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️ Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/web-universe-main-dev/worker-01/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
🔐 Creating replicator role for worker-01...
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
NAME : web-universe-main-dev
IDENTIFIER : worker-01
PG HOST : db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432
ROLE : replicator
SSL DIR : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01
DNS → 10.100.1.243
CA : /home/www-data/ssl/.postgresql/web-universe-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️ Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE: Creating role: replicator with password
SET
CREATE ROLE
✅ Replicator role ensured with password authentication.
ℹ️ Password stored in: AWS Secrets Manager
Secret name: fastorder/db/web/universe/main/dev/postgresql/replicator
🔄 MIGRATION PATH: Password → Certificate Authentication
Current: SCRAM-SHA-256 password auth (production-ready)
Future: Certificate-based auth (requires CA automation)
To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
and configure standby to use SSL certificates instead of password
🎉 Done.
✅ Replicator role created for worker-01
[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=da95d2d5-7a04-4279-ae1c-c482185a88f1)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
ℹ️ Service-specific setup (web) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)
🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=web
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
🔍 DEBUG_CHECKPOINT_03: Found service-specific run.sh for: web
[DEBUG] Tracking substep start: steps/01-install/steps/web (RUN_UUID=da95d2d5-7a04-4279-ae1c-c482185a88f1)
[INFO] 🔸 Service: web (service-specific)
🔍 DEBUG_CHECKPOINT_04: Executing service-specific: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/web/run.sh
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Web Service Schema Provisioning
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service: web
[INFO] Zone: universe
[INFO] Branch: main
[INFO] Environment: dev
[INFO] Identifier: coordinator
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] 📦 Running: 01-init-schema.sh...
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Initializing config schema
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Identifier: coordinator
Database: fastorder_web_universe_main_dev_db
Host: db-web-universe-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking if config schema exists...
[INFO] Config schema already exists - checking tables...
[OK] Config schema with 3 tables already exists - skipping
[OK] ✓ 01-init-schema.sh completed
[INFO] ═══════════════════════════════════════════════════════════════
[OK] Web service schema provisioning complete
[INFO] ═══════════════════════════════════════════════════════════════
[0;32m✓[0m ✅ Worker worker-01 setup completed
Skipping standbys (PG_WORKERS_STANDBY_NUM=0)
[0;32m✓[0m ✅ PostgreSQL installation completed
[0;34m[INFO][0m Discovering additional setup steps...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 02-pg-bouncer.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up PgBouncer connection pooling...
[2026-02-05 06:42:29 UTC] USER=www-data EUID=0 PID=4189306 ACTION=fsop ARGS=rm -f /tmp/pgbouncer-ip.service /tmp/pgbouncer.service
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;32m✓ [SECRETS][0m Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[0;34m[SECRETS][0m Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[0;34m[SECRETS][0m Search (build_es_secret_name, get/set_es_credentials_to_vault)
[0;34m[SECRETS][0m Backups (build_backup_path)
[0;34m[SECRETS][0m Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[0;34m[INFO][0m Checking for existing PgBouncer application environment in topology …
[0;32m[OK][0m Using existing PgBouncer environment:
[0;34m[INFO][0m IP: 10.100.1.244
[0;34m[INFO][0m FQDN: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[0;34m[INFO][0m Domain: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[0;34m[INFO][0m Ensuring /etc/hosts entry for db-web-universe-main-dev-postgresql-bouncer.fastorder.com …
[0;32m[OK][0m /etc/hosts already contains entry for db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[1;33m[WARN][0m IP 10.100.1.244 is assigned to multiple interfaces:
inet 10.100.1.242/32 scope global lo
valid_lft forever preferred_lft forever
inet 10.100.1.244/32 scope global lo:pgbouncer
--
inet 10.100.1.243/32 scope global eth0
valid_lft forever preferred_lft forever
inet 10.100.1.244/32 scope global eth0:244
[1;33m[WARN][0m This may cause routing issues
[0;34m[INFO][0m Final verification of /etc/hosts entry for db-web-universe-main-dev-postgresql-bouncer.fastorder.com …
[0;32m[OK][0m /etc/hosts correctly maps db-web-universe-main-dev-postgresql-bouncer.fastorder.com to 10.100.1.244
[0;32m[OK][0m PgBouncer IP 10.100.1.244 already correctly bound to lo:pgbouncer
[2026-02-05 06:42:30 UTC] USER=www-data EUID=0 PID=4189408 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 06:42:31 UTC] USER=www-data EUID=0 PID=4189514 ACTION=passthru ARGS=systemctl restart pgbouncer-ip@web-universe-main-dev.service
[2026-02-05 06:42:31 UTC] USER=www-data EUID=0 PID=4189544 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer-ip@web-universe-main-dev.service
[1;33m[WARN][0m pgbouncer-ip@web-universe-main-dev.service is not active
[1;33m[WARN][0m Check status: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@web-universe-main-dev.service
[2026-02-05 06:42:32 UTC] USER=www-data EUID=0 PID=4189667 ACTION=fsop ARGS=mkdir -p /etc/pgbouncer/web-universe-main-dev
[2026-02-05 06:42:32 UTC] USER=www-data EUID=0 PID=4189677 ACTION=fsop ARGS=mkdir -p /run/pgbouncer/web-universe-main-dev
[2026-02-05 06:42:32 UTC] USER=www-data EUID=0 PID=4189686 ACTION=fsop ARGS=mkdir -p /var/log/pgbouncer/web-universe-main-dev
[2026-02-05 06:42:32 UTC] USER=www-data EUID=0 PID=4189695 ACTION=fsop ARGS=chmod 750 /etc/pgbouncer/web-universe-main-dev
[2026-02-05 06:42:32 UTC] USER=www-data EUID=0 PID=4189704 ACTION=fsop ARGS=chmod 750 /run/pgbouncer/web-universe-main-dev
[2026-02-05 06:42:32 UTC] USER=www-data EUID=0 PID=4189713 ACTION=fsop ARGS=chmod 750 /var/log/pgbouncer/web-universe-main-dev
[2026-02-05 06:42:32 UTC] USER=www-data EUID=0 PID=4189728 ACTION=fsop ARGS=chown root:postgres /etc/pgbouncer/web-universe-main-dev
[2026-02-05 06:42:32 UTC] USER=www-data EUID=0 PID=4189737 ACTION=fsop ARGS=chown postgres:postgres /run/pgbouncer/web-universe-main-dev
[2026-02-05 06:42:32 UTC] USER=www-data EUID=0 PID=4189747 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbouncer/web-universe-main-dev
[0;34m[INFO][0m Generating pgbouncer_admin client certificates...
[0;34m[INFO][0m ⏳ This may take 30-60 seconds...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
Environment: web-universe-main-dev
Username: pgbouncer_admin
Identifier: pgbouncer
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: web-universe-main-dev
Service: web
Zone: universe
Branch: main
Env: dev
Node: pgbouncer
User (CN): pgbouncer_admin
Hostname: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4189820 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-pgbouncer-pgbouncer_admin
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4189829 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4189840 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4189859 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = pgbouncer_admin
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4189874 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4189885 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4189896 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4189916 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4189925 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4189934 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key.pkcs1 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4189943 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_der.key /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4189952 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_pk8.der /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4189962 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4189971 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4189980 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4189990 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4189999 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4190008 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4190017 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4190026 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4190036 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4190045 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4190054 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 06:42:33 UTC] USER=www-data EUID=0 PID=4190063 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190089 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190108 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190117 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190129 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190161 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190170 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190179 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190188 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190197 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190217 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190226 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190238 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190247 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190258 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190267 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190276 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190285 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190294 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190306 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190316 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190325 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190335 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190345 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:34 UTC] USER=www-data EUID=0 PID=4190354 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190363 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190372 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190381 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190399 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190408 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190417 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190426 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190435 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190444 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190454 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190464 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190473 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190482 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190492 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190501 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190513 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190524 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190533 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190546 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190558 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190567 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190576 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190588 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190598 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190607 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:42:35 UTC] USER=www-data EUID=0 PID=4190616 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/web-universe-main-dev/ca.pem
✅ Symlinked ca.pem
✅ Symlinked client-cert.pem
✅ Symlinked client-key.pem
[2026-02-05 06:42:36 UTC] USER=www-data EUID=0 PID=4190643 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/web-universe-main-dev
[2026-02-05 06:42:36 UTC] USER=www-data EUID=0 PID=4190652 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 06:42:36 UTC] USER=www-data EUID=0 PID=4190661 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/web-universe-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/web-universe-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/web-universe-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: web-universe-main-dev
User: pgbouncer_admin
Node: pgbouncer
FQDN: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
database.sslkey: /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/web-universe-main-dev/pgbouncer/pgbouncer_admin.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/web-universe-main-dev/pgbouncer/root.crt"
export PGSSLMODE="verify-full"
psql -h db-web-universe-main-dev-postgresql-bouncer.fastorder.com -U pgbouncer_admin -d postgres
[0;32m[OK][0m mTLS client certificate present: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[0;34m[INFO][0m Creating symlinks to canonical certificates in /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend...
[2026-02-05 06:42:36 UTC] USER=www-data EUID=0 PID=4190687 ACTION=fsop ARGS=mkdir -p /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend
[2026-02-05 06:42:36 UTC] USER=www-data EUID=0 PID=4190696 ACTION=fsop ARGS=mkdir -p /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend
[2026-02-05 06:42:36 UTC] USER=www-data EUID=0 PID=4190705 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-02-05 06:42:36 UTC] USER=www-data EUID=0 PID=4190723 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt
[0;34m[INFO][0m Creating coordinator CA symlink for PostgreSQL server verification...
[2026-02-05 06:42:36 UTC] USER=www-data EUID=0 PID=4190732 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m Verifying canonical certificate permissions...
[2026-02-05 06:42:36 UTC] USER=www-data EUID=0 PID=4190750 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-02-05 06:42:36 UTC] USER=www-data EUID=0 PID=4190762 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt
[2026-02-05 06:42:36 UTC] USER=www-data EUID=0 PID=4190773 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key
[0;32m[OK][0m Backend certificate symlinks created in /etc/ssl
[0;32m[OK][0m Coordinator CA symlink created for server verification
[0;32m[OK][0m Certificates already in canonical location - no symlinks needed
[2026-02-05 06:42:36 UTC] USER=www-data EUID=0 PID=4190786 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
[2026-02-05 06:42:36 UTC] USER=www-data EUID=0 PID=4190795 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
[2026-02-05 06:42:36 UTC] USER=www-data EUID=0 PID=4190804 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt
[0;34m[INFO][0m PgBouncer will use PostgreSQL coordinator CA: /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;32m[OK][0m PostgreSQL coordinator at db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432 is reachable
[0;34m[INFO][0m Dumping SCRAM secrets from coordinator for PgBouncer auth_file …
[2026-02-05 06:42:36 UTC] USER=www-data EUID=0 PID=4190834 ACTION=fsop ARGS=cp /tmp/tmp.3xCWRJmpDw /etc/pgbouncer/web-universe-main-dev/userlist.txt
[2026-02-05 06:42:36 UTC] USER=www-data EUID=0 PID=4190843 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/userlist.txt
[2026-02-05 06:42:36 UTC] USER=www-data EUID=0 PID=4190867 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;32m[OK][0m Auth file written: /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;34m[INFO][0m Retrieved password from vault for pgbouncer_admin
[0;34m[INFO][0m Ensuring PgBouncer admin role 'pgbouncer_admin' exists in Postgres (coordinator) …
[0;32m[OK][0m Role pgbouncer_admin created/updated successfully
[0;34m[SECRETS][0m Setting credentials in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/pgbouncer_admin
[0;32m✓ [SECRETS][0m Credentials updated in vault: fastorder/db/web/universe/main/dev/postgresql/coordinator/pgbouncer_admin
[0;34m[INFO][0m ✅ PgBouncer admin password stored in centralized secrets vault
[0;34m[INFO][0m Re-fetching SCRAM secrets after role creation to ensure pgbouncer_admin is included …
[2026-02-05 06:42:42 UTC] USER=www-data EUID=0 PID=4191048 ACTION=fsop ARGS=cp /tmp/tmp.HxmIl8XKUg /etc/pgbouncer/web-universe-main-dev/userlist.txt
[2026-02-05 06:42:42 UTC] USER=www-data EUID=0 PID=4191057 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/userlist.txt
[2026-02-05 06:42:42 UTC] USER=www-data EUID=0 PID=4191075 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;32m[OK][0m Auth file updated with pgbouncer_admin SCRAM hash
[0;34m[INFO][0m Auth file contains [2026-02-05 06:42:42 UTC] USER=www-data EUID=0 PID=4191089 ACTION=passthru ARGS=bash -c wc -l < '/etc/pgbouncer/web-universe-main-dev/userlist.txt'
4 user(s)
[0;32m[OK][0m Admin 'pgbouncer_admin' password generated and saved
[0;34m[INFO][0m Configuring PostgreSQL to prevent Citus metadata sync hangs...
ALTER ROLE
[0;32m[OK][0m Disabled Citus metadata sync for pgbouncer_admin
[0;34m[INFO][0m Verifying application database fastorder_web_universe_main_dev_db exists...
[0;32m[OK][0m ✓ Database fastorder_web_universe_main_dev_db exists
[0;34m[INFO][0m Granting permissions to pgbouncer_admin on fastorder_web_universe_main_dev_db...
GRANT
[0;32m[OK][0m ✓ Granted CONNECT on fastorder_web_universe_main_dev_db to pgbouncer_admin
GRANT
[0;32m[OK][0m ✓ Granted USAGE on schema public to pgbouncer_admin
GRANT
[0;32m[OK][0m ✓ Granted SELECT on all tables to pgbouncer_admin
ALTER DATABASE
[0;32m[OK][0m Set synchronous_commit=local for fastorder_web_universe_main_dev_db
[0;34m[INFO][0m Ensuring pg_hba.conf entry for pgbouncer_admin …
[0;34m[INFO][0m Adding pg_hba.conf entries for pgbouncer_admin with cert auth …
[0;32m[OK][0m pg_hba.conf updated and PostgreSQL configuration reloaded
[2026-02-05 06:42:43 UTC] USER=unknown EUID=33 PID=4191151 ACTION=-u ARGS=postgres bash
ERROR: Invalid or unauthorized action: -u
[1;33m[WARN][0m pg_hba.conf entry may not have loaded correctly
[0;34m[INFO][0m Writing /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini …
[2026-02-05 06:42:44 UTC] USER=www-data EUID=0 PID=4191202 ACTION=fsop ARGS=cp /tmp/tmp.919y6jd5uO /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
[2026-02-05 06:42:44 UTC] USER=www-data EUID=0 PID=4191211 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
[2026-02-05 06:42:44 UTC] USER=www-data EUID=0 PID=4191220 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
[2026-02-05 06:42:45 UTC] USER=www-data EUID=0 PID=4191229 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbouncer/web-universe-main-dev /run/pgbouncer/web-universe-main-dev /var/log/pgbouncer/web-universe-main-dev
[2026-02-05 06:42:45 UTC] USER=www-data EUID=0 PID=4191238 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/web-universe-main-dev/userlist.txt
[0;32m[OK][0m pgbouncer.ini ready
[0;34m[INFO][0m Verifying TLS settings in pgbouncer.ini:
[2026-02-05 06:42:45 UTC] USER=www-data EUID=0 PID=4191248 ACTION=fsop ARGS=grep -E (client_tls_sslmode|server_tls) /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
client_tls_sslmode = verify-full
server_tls_sslmode = verify-full
server_tls_ca_file = /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
server_tls_cert_file = /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
server_tls_key_file = /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;34m[INFO][0m Verifying PgBouncer server certificate files:
[2026-02-05 06:42:45 UTC] USER=www-data EUID=0 PID=4191257 ACTION=fsop ARGS=test -r /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[0;32m[OK][0m Server cert readable by postgres: /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-02-05 06:42:45 UTC] USER=www-data EUID=0 PID=4191267 ACTION=fsop ARGS=test -r /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;32m[OK][0m Server key readable by postgres: /etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;34m[INFO][0m Verifying coordinator CA certificate:
[2026-02-05 06:42:45 UTC] USER=www-data EUID=0 PID=4191276 ACTION=fsop ARGS=test -r /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;32m[OK][0m Coordinator CA readable by postgres: /etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m Preflight: stopping any conflicting PgBouncer on 6432 …
[2026-02-05 06:42:45 UTC] USER=www-data EUID=0 PID=4191294 ACTION=passthru ARGS=systemctl stop pgbouncer@web-universe-main-dev.service
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json?all=1": dial unix /var/run/docker.sock: connect: permission denied
[2026-02-05 06:42:47 UTC] USER=www-data EUID=0 PID=4191401 ACTION=passthru ARGS=systemctl daemon-reload
[0;32m[OK][0m systemd unit installed: pgbouncer@web-universe-main-dev.service
[0;34m[INFO][0m Running pre-flight IP conflict check for 10.100.1.244:6432 …
[1;33m[WARN][0m IP conflict checker not found at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/check-ip-conflicts.sh
[1;33m[WARN][0m Skipping pre-flight check - conflicts may occur
[0;34m[INFO][0m Starting PgBouncer (web-universe-main-dev) …
[2026-02-05 06:42:48 UTC] USER=www-data EUID=0 PID=4191514 ACTION=passthru ARGS=systemctl restart pgbouncer@web-universe-main-dev.service
[2026-02-05 06:42:48 UTC] USER=www-data EUID=0 PID=4191532 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer@web-universe-main-dev.service
[0;32m[OK][0m Service ACTIVE
[0;34m[INFO][0m Verifying auth_file before probing …
[0;34m[INFO][0m Auth file contains 4 user(s)
[1;33m[WARN][0m Auth file does NOT contain pgbouncer_admin entry - authentication will fail
[0;34m[INFO][0m Probing admin console via SSL (psql to database 'pgbouncer') …
[0;34m[INFO][0m Retrieved password from vault for admin console probe
[0;32m[OK][0m Admin console reachable (SHOW POOLS OK)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Running Comprehensive PgBouncer Verification Tests
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Password extracted: WO0D2C0d7Z... (using postgres user certificates)
[0;34m[INFO][0m Test 1/7: Admin Console - SHOW POOLS
database | user | cl_active | cl_waiting | cl_active_cancel_req | cl_waiting_cancel_req | sv_active | sv_active_cancel | sv_being_canceled | sv_idle | sv_used | sv_tested | sv_login | maxwait | maxwait_us | pool_mode | load_balance_hosts
-----------+-----------+-----------+------------+----------------------+-----------------------+-----------+------------------+-------------------+---------+---------+-----------+----------+---------+------------+-----------+--------------------
pgbouncer | pgbouncer | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | statement |
(1 row)
[0;32m[OK][0m ✓ SHOW POOLS: SUCCESS
[0;34m[INFO][0m Test 2/7: Admin Console - SHOW VERSION
[0;32m[OK][0m ✓ SHOW VERSION: PgBouncer 1.24.1
[0;34m[INFO][0m Test 3/7: Admin Console - SHOW STATS
database | total_server_assignment_count | total_xact_count | total_query_count | total_received | total_sent | total_xact_time | total_query_time | total_wait_time | total_client_parse_count | total_server_parse_count | total_bind_count | avg_server_assignment_count | avg_xact_count | avg_query_count | avg_recv | avg_sent | avg_xact_time | avg_query_time | avg_wait_time | avg_client_parse_count | avg_server_parse_count | avg_bind_count
-----------+-------------------------------+------------------+-------------------+----------------+------------+-----------------+------------------+-----------------+--------------------------+--------------------------+------------------+-----------------------------+----------------+-----------------+----------+----------+---------------+----------------+---------------+------------------------+------------------------+----------------
pgbouncer | 0 | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
(1 row)
[0;32m[OK][0m ✓ SHOW STATS: SUCCESS
[0;34m[INFO][0m Test 4/7: Admin Console - SHOW DATABASES
name | host | port | database | force_user | pool_size | min_pool_size | reserve_pool_size | server_lifetime | pool_mode | load_balance_hosts | max_connections | current_connections | max_client_connections | current_client_connections | paused | disabled
---------------------------------------------+---------------------------------------------------------------+------+------------------------------------+------------+-----------+---------------+-------------------+-----------------+-----------+--------------------+-----------------+---------------------+------------------------+----------------------------+--------+----------
fastorder_web_universe_main_dev_db | db-web-universe-main-dev-postgresql-coordinator.fastorder.com | 5432 | fastorder_web_universe_main_dev_db | | 100 | 0 | 20 | 3600 | | | 0 | 0 | 0 | 0 | 0 | 0
fastorder_web_universe_main_dev_db_worker_1 | pg-worker-01-web-universe-main-dev.fastorder.com | 5432 | fastorder_web_universe_main_dev_db | | 100 | 0 | 20 | 3600 | | | 0 | 0 | 0 | 0 | 0 | 0
fastorder_web_universe_main_dev_db_worker_2 | pg-worker-01-standby-01-web-universe-main-dev.fastorder.com | 5432 | fastorder_web_universe_main_dev_db | | 100 | 0 | 20 | 3600 | | | 0 | 0 | 0 | 0 | 0 | 0
fastorder_web_universe_main_dev_db_worker_3 | pg-worker-01-standby-02-web-universe-main-dev.fastorder.com | 5432 | fastorder_web_universe_main_dev_db | | 100 | 0 | 20 | 3600 | | | 0 | 0 | 0 | 0 | 0 | 0
fastorder_web_universe_main_dev_db_worker_4 | pg-worker-01-standby-03-web-universe-main-dev.fastorder.com | 5432 | fastorder_web_universe_main_dev_db | | 100 | 0 | 20 | 3600 | | | 0 | 0 | 0 | 0 | 0 | 0
pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | 0 | 3600 | statement | | 0 | 0 | 0 | 1 | 0 | 0
(6 rows)
[0;32m[OK][0m ✓ SHOW DATABASES: SUCCESS
[0;34m[INFO][0m Test 5/7: Admin Console - SHOW CONFIG
[0;32m[OK][0m ✓ SHOW CONFIG: SUCCESS
[0;34m[INFO][0m Key settings:
[0;34m[INFO][0m client_tls_sslmode = verify-full|disable|yes
[0;34m[INFO][0m max_client_conn = 2048|100|yes
[0;34m[INFO][0m pool_mode = transaction|session|yes
[0;34m[INFO][0m server_tls_sslmode = verify-full|prefer|yes
psql "host=db-web-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_web_universe_main_dev_db user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD connect_timeout=5 sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/root.crt sslcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.crt sslkey=/etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/pgbouncer_admin.key" --no-psqlrc -Atc 'SELECT version();'
[0;34m[INFO][0m Test 6/7: Application Database - SELECT version()
[1;33m[WARN][0m ✗ Application database query: FAILED (timeout or connection issue)
[1;33m[WARN][0m If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[0;34m[INFO][0m Test 7/8: Application Database - Connection Details
[1;33m[WARN][0m ✗ Connection details: FAILED (timeout or connection issue)
[1;33m[WARN][0m If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[0;34m[INFO][0m Test 8/8: End-to-End Application Routing - Pool Verification
[0;34m[INFO][0m Running actual queries through PgBouncer to verify routing and pooling...
[1;33m[WARN][0m ✗ End-to-end routing verification: FAILED - All 3 queries failed
[1;33m[WARN][0m If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[1;33m[WARN][0m Otherwise check if database fastorder_web_universe_main_dev_db exists and user pgbouncer_admin has permissions
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verification Complete - Tests 1-5 PASSED (Admin console verified)
[1;33m[WARN][0m Tests 6-8 FAILED - Application database not accessible
[1;33m[WARN][0m This is expected if Citus is not set up yet
[1;33m[WARN][0m Run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m PgBouncer is up for web-universe-main-dev
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Connection Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Password stored in: AWS Secrets Manager (fastorder/db/web/ksa/main/dev/postgresqlweb/universe/main/dev/coordinator-pgbouncer_admin)
Current password: WO0D2C0d7ZbIdk65D10y9TaD
1. Admin Console (using IP address to avoid DNS/SSL issues):
psql "host=10.100.1.244 port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD sslmode=verify-full sslrootcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"
2. Admin Console (using hostname):
psql "host=db-web-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD sslmode=verify-full sslrootcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"
3. Application Database:
psql "host=db-web-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_web_universe_main_dev_db sslkey=/etc/ssl/private/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=WO0D2C0d7ZbIdk65D10y9TaD sslmode=verify-full sslrootcert=/etc/ssl/certs/web-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"
4. Using .pgpass file:
echo "db-web-universe-main-dev-postgresql-bouncer.fastorder.com:6432:*:pgbouncer_admin:WO0D2C0d7ZbIdk65D10y9TaD" >> ~/.pgpass
chmod 600 ~/.pgpass
psql -h db-web-universe-main-dev-postgresql-bouncer.fastorder.com -p 6432 -U pgbouncer_admin -d fastorder_web_universe_main_dev_db
5. Retrieve password from vault:
source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
PGPASSWORD="$(get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password')" \
psql -h 10.100.1.244 -p 6432 -U pgbouncer_admin -d pgbouncer -c "SHOW POOLS;"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Architecture
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
• Default db 'fastorder_web_universe_main_dev_db' → Citus coordinator (db-web-universe-main-dev-postgresql-coordinator.fastorder.com)
• Worker access: 'fastorder_web_universe_main_dev_db_worker_1', 'fastorder_web_universe_main_dev_db_worker_2', … (if exist)
• Client TLS: require (password auth) / verify-full (mTLS with certs)
• Server TLS: verify-full (PgBouncer validates PostgreSQL certs)
• Auth: SCRAM-SHA-256 via /etc/pgbouncer/web-universe-main-dev/userlist.txt
• Pool mode: transaction (stateless connections)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Management
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Service Status:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer@web-universe-main-dev.service
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@web-universe-main-dev.service
Logs:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@web-universe-main-dev.service -f
/usr/local/bin/fastorder-provisioning-wrapper.sh tail -f /var/log/pgbouncer/web-universe-main-dev/pgbouncer.log
Reload Config:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@web-universe-main-dev.service
Restart:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@web-universe-main-dev.service
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Files
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Config: /etc/pgbouncer/web-universe-main-dev/pgbouncer.ini
Auth file: /etc/pgbouncer/web-universe-main-dev/userlist.txt
Server cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.crt
Server key: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/server.key
CA cert: /etc/fastorder/postgresql/certs/web-universe-main-dev/pgbouncer/ca.crt
PG CA: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/ca.crt
Logs: /var/log/pgbouncer/web-universe-main-dev/pgbouncer.log
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Troubleshooting
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
If "SASL authentication failed":
1. Check auth file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/web-universe-main-dev/userlist.txt
2. Verify pgbouncer_admin is present with SCRAM hash
3. Get password from vault:
source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password'
4. Reload PgBouncer: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@web-universe-main-dev.service
If "no pg_hba.conf entry":
1. Check pg_hba.conf on coordinator
2. Add rule: hostssl all pgbouncer_admin 10.100.1.244/32 cert clientcert=verify-full
3. Reload PostgreSQL
To add users to PgBouncer:
1. Create user in PostgreSQL with password
2. Re-run SCRAM dump:
psql "host=db-web-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 dbname=postgres user=postgres \
sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/root.crt \
sslcert=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.crt sslkey=/etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/postgres.key" \
-Atc "SELECT '\"' || rolname || '\" \"' || rolpassword || '\"' \
FROM pg_authid WHERE rolpassword LIKE 'SCRAM-SHA-256%' \
AND rolcanlogin ORDER BY rolname;" | command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop tee /etc/pgbouncer/web-universe-main-dev/userlist.txt
3. Reload: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@web-universe-main-dev.service
[0;34m[INFO][0m Registering PgBouncer node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: PgBouncer
[INFO] Identifier: web-universe-main-dev-pgbouncer
[INFO] Identifier Parent: postgresql
[INFO] IP: 10.100.1.244
[INFO] Port: 6432
[INFO] FQDN: db-web-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: fdc70c5f-615d-432f-8161-a7acd56ea9ed
[SUCCESS] Environment UUID: b9b5805b-8033-4d13-b5de-5df5c0dd562b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b9b5805b-8033-4d13-b5de-5df5c0dd562b
[0;32m[OK][0m PgBouncer node registered to observability API
[0;32m✓[0m ✅ PgBouncer setup completed
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 03-citus-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m CITUS DISTRIBUTED CLUSTER SETUP
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Phase 1: Installing Citus extension on workers...
[0;34m[INFO][0m Phase 2: Setting up coordinator and registering workers...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 📦 PHASE 1: Installing Citus extension on 1 worker(s)...
[0;34m[INFO][0m → Worker 1/1: Installing Citus on worker-01...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS CLUSTER SETUP
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m 🔧 Setting up Citus Worker...
[0;34m[INFO][0m Temporarily disabling synchronous replication for extension installation...
t
[0;34m[INFO][0m Installing Citus extension on worker...
[0;32m[OK][0m Citus extension installed on worker
[0;34m[INFO][0m Restoring synchronous replication settings...
t
[0;34m[INFO][0m Worker Citus extension installed - registration will happen when coordinator setup runs
[0;32m[OK][0m Citus setup complete for worker-01
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m✓[0m ✅ Citus extension installed on worker-01
[0;32m✓[0m ✅ Phase 1 Complete: All 1 workers have Citus extension installed
[0;34m[INFO][0m 🔧 PHASE 2: Setting up Citus coordinator and registering workers...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS CLUSTER SETUP
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m 🔧 Setting up Citus Coordinator...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m DIAGNOSTIC: Configuration Variables
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m PG_WORKERS_NUM: 1
[0;34m[INFO][0m ENV_ID: web-universe-main-dev
[0;34m[INFO][0m DOMAIN: fastorder.com
[0;34m[INFO][0m PORT: 5432
[0;34m[INFO][0m SOCKET_DIR: /var/run/postgresql-web-universe-main-dev-coordinator
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Ensuring postgres client certificates exist for coordinator...
[0;32m[OK][0m Postgres client certificates already exist for coordinator
[0;34m[INFO][0m Adding citus_cert_map to coordinator pg_ident.conf...
[0;32m[OK][0m pg_ident.conf updated for coordinator
[0;34m[INFO][0m Installing Citus extension on coordinator...
[0;32m[OK][0m Citus extension installed on coordinator (postgres database)
[0;34m[INFO][0m Installing Citus extension on application database: fastorder_web_universe_main_dev_db...
[0;32m[OK][0m Citus extension installed on application database: fastorder_web_universe_main_dev_db
[0;34m[INFO][0m Configuring Citus SSL connection parameters...
[2026-02-05 06:43:19 UTC] USER=www-data EUID=0 PID=4192712 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m ✅ Citus SSL connection parameters configured: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator
[1;33m[WARN][0m Node not identified as coordinator, initializing...
[0;34m[INFO][0m Checking coordinator configuration...
[0;34m[INFO][0m Persisting citus.local_hostname to postgresql.conf...
[2026-02-05 06:43:21 UTC] USER=www-data EUID=0 PID=4192805 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/web-universe-main-dev/coordinator/postgresql.conf
[2026-02-05 06:43:22 UTC] USER=www-data EUID=0 PID=4192829 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
[0;32m[OK][0m ✅ citus.local_hostname persisted to config and reloaded
[0;34m[INFO][0m Configuring coordinator hostname in postgres database: db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432
[0;32m[OK][0m ✅ Coordinator hostname set to db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432 in postgres database
[0;34m[INFO][0m Checking coordinator configuration in application database: fastorder_web_universe_main_dev_db...
[0;34m[INFO][0m Configuring coordinator hostname in application database: db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432
[0;32m[OK][0m ✅ Coordinator hostname set to db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432 in application database
[0;34m[INFO][0m Validating coordinator configuration before worker registration...
[0;32m[OK][0m ✅ Coordinator hostname validated: db-web-universe-main-dev-postgresql-coordinator.fastorder.com
[0;32m[OK][0m ✅ citus_tables view is accessible
[0;34m[INFO][0m Checking coordinator self-registration...
[0;32m[OK][0m ✅ Coordinator is already self-registered
[0;34m[INFO][0m Configuring coordinator shard placement policy...
[0;32m[OK][0m ✅ Coordinator already configured in postgres database (shouldhaveshards = false)
[0;32m[OK][0m ✅ Coordinator already configured in application database (shouldhaveshards = false)
[0;34m[INFO][0m Registering 1 worker(s) to Citus cluster...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m PRE-FLIGHT: Checking worker availability...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Checking worker worker-01...
[0;34m[INFO][0m FQDN: db-web-universe-main-dev-postgresql-worker-01.fastorder.com
[0;32m[OK][0m ✅ Worker worker-01 is reachable via SSL
[0;32m[OK][0m All workers are reachable - proceeding with registration
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Adding Citus worker: db-web-universe-main-dev-postgresql-worker-01.fastorder.com:5432
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Adding citus_cert_map to worker-01 pg_ident.conf...
[0;32m[OK][0m pg_ident.conf updated for worker-01
[0;34m[INFO][0m Configuring worker worker-01 HBA for coordinator (10.100.1.54) access...
[0;32m[OK][0m Worker worker-01 HBA configured for coordinator (10.100.1.54)
[0;34m[INFO][0m Adding replication rules for 3 standby(s)...
[0;32m[OK][0m Replication rules added for worker-01
[0;34m[INFO][0m Reloading worker worker-01 to apply HBA changes...
[2026-02-05 06:43:25 UTC] USER=www-data EUID=0 PID=4193038 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-worker-01.service
[0;34m[INFO][0m Configuring coordinator HBA for worker worker-01 (10.100.1.243) access...
[0;32m[OK][0m Coordinator HBA configured for worker worker-01 (10.100.1.243)
[0;34m[INFO][0m Reloading coordinator to apply HBA changes...
[2026-02-05 06:43:25 UTC] USER=www-data EUID=0 PID=4193069 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-coordinator.service
[0;34m[INFO][0m Ensuring postgres client certificates exist for worker-01...
[0;32m[OK][0m Postgres client certificates already exist for worker-01
[0;34m[INFO][0m Configuring citus.node_conninfo on worker-01...
[2026-02-05 06:43:25 UTC] USER=www-data EUID=0 PID=4193087 ACTION=passthru ARGS=systemctl reload postgresql@web-universe-main-dev-worker-01.service
[0;32m[OK][0m citus.node_conninfo configured on worker-01
[0;34m[INFO][0m Temporarily relaxing sync-rep on worker worker-01...
t
[0;32m[OK][0m Worker worker-01 sync-rep relaxed (was: sync_commit=on)
[0;34m[INFO][0m Ensuring Citus extension on worker databases...
CREATE EXTENSION
CREATE EXTENSION
[0;34m[INFO][0m Running citus_add_node with 180s timeout...
2
[0;34m[INFO][0m Restoring worker worker-01 sync-rep settings...
t
[0;32m[OK][0m Worker worker-01 sync-rep restored
[0;32m[OK][0m ✅ Worker db-web-universe-main-dev-postgresql-worker-01.fastorder.com successfully added to Citus cluster
[0;34m[INFO][0m Node ID: 2
[0;34m[INFO][0m Registered in: postgres, fastorder_web_universe_main_dev_db
[0;32m[OK][0m Worker worker-01 registration successful
[0;34m[INFO][0m Configuring worker worker-01 shard placement policy...
[0;32m[OK][0m ✅ Worker worker-01 configured to hold shards in all databases
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m POST-REGISTRATION: Verifying cluster state...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Expected workers: 1
[0;34m[INFO][0m Registered workers: 1
[0;32m[OK][0m ✅ All 1 workers successfully registered!
[0;34m[INFO][0m Citus cluster configuration:
db-web-universe-main-dev-postgresql-coordinator.fastorder.com 5432 0 t primary f
db-web-universe-main-dev-postgresql-worker-01.fastorder.com 5432 1 t primary t
[0;34m[INFO][0m Note: groupid=0 is the coordinator, groupid>0 are workers
[0;34m[INFO][0m shouldhaveshards: false=query router only, true=holds data shards
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m FINAL VALIDATION: Verifying configuration persistence...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 06:43:30 UTC] USER=www-data EUID=0 PID=4193383 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/web-universe-main-dev/coordinator/postgresql.conf
[0;32m[OK][0m ✅ citus.local_hostname persisted in postgresql.conf
[0;32m[OK][0m ✅ All 1 worker(s) successfully registered and verified
[0;32m[OK][0m ✅ All validation checks passed
[0;32m[OK][0m Citus coordinator setup complete
[0;32m[OK][0m Citus setup complete for coordinator
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m✓[0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✓[0m ✅ CITUS CLUSTER SETUP COMPLETED SUCCESSFULLY
[0;32m✓[0m Coordinator: Ready and accepting connections
[0;32m✓[0m Workers registered: 1
[0;32m✓[0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 05-backup-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up coordinator backup...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for web-universe-main-dev...
[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO] Version: pgBackRest 2.56.0
[INFO] 2️⃣ Creating backup directories...
[2026-02-05 06:43:32 UTC] USER=www-data EUID=0 PID=4193596 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 06:43:32 UTC] USER=www-data EUID=0 PID=4193610 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/web-universe-main-dev
[2026-02-05 06:43:32 UTC] USER=www-data EUID=0 PID=4193622 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-02-05 06:43:32 UTC] USER=www-data EUID=0 PID=4193631 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-02-05 06:43:32 UTC] USER=www-data EUID=0 PID=4193649 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-02-05 06:43:34 UTC] USER=www-data EUID=0 PID=4193713 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-02-05 06:43:34 UTC] USER=www-data EUID=0 PID=4193725 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-02-05 06:43:34 UTC] USER=www-data EUID=0 PID=4193735 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-02-05 06:43:34 UTC] USER=www-data EUID=0 PID=4193744 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 06:43:34 UTC] USER=www-data EUID=0 PID=4193753 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO] ✅ Backup directories created
[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-web-universe-main-dev
[2026-02-05 06:43:35 UTC] USER=www-data EUID=0 PID=4193774 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-02-05 06:43:35 UTC] USER=www-data EUID=0 PID=4193783 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key
[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-02-05 06:43:35 UTC] USER=www-data EUID=0 PID=4193792 ACTION=fsop ARGS=find /data/postgresql/17/web-universe-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-02-05 06:43:35 UTC] USER=www-data EUID=0 PID=4193801 ACTION=fsop ARGS=chown -R postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed
[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-02-05 06:43:35 UTC] USER=www-data EUID=0 PID=4193810 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-02-05 06:43:35 UTC] USER=www-data EUID=0 PID=4193819 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-02-05 06:43:35 UTC] USER=www-data EUID=0 PID=4193828 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created
[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-02-05 06:43:35 UTC] USER=www-data EUID=0 PID=4193837 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/coordinator/PG_VERSION
[2026-02-05 06:43:35 UTC] USER=www-data EUID=0 PID=4193847 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running
[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza web-universe-main-dev-coordinator already initialized and verified
[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
[INFO] ✅ WAL archiving configured for coordinator
[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-02-05 06:43:35 UTC] USER=www-data EUID=0 PID=4193904 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-02-05 06:43:37 UTC] USER=www-data EUID=0 PID=4193945 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 06:43:41 UTC] USER=www-data EUID=0 PID=4194080 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled
[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-02-05 06:43:41 UTC] USER=www-data EUID=0 PID=4194109 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator --log-level-console=info check
2026-02-05 06:43:41.949 P00 INFO: check command begin 2.56.0: --exec-id=4194117-2e92452b --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 06:43:41.968 P00 INFO: check repo1 configuration (primary)
2026-02-05 06:43:41.980 P00 ERROR: [028]: backup and archive info files exist but do not match the database
HINT: is this the correct stanza?
HINT: did an error occur during stanza-upgrade?
2026-02-05 06:43:41.980 P00 INFO: check command end: aborted with exception [028]
[WARN] ⚠️ Stanza verification failed - this may be normal if WAL archiving hasn't started yet
[WARN] The backup system is configured and will work once WAL segments are generated
[INFO] 8️⃣ Creating backup automation scripts...
[2026-02-05 06:43:42 UTC] USER=www-data EUID=0 PID=4194130 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 06:43:42 UTC] USER=www-data EUID=0 PID=4194139 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 06:43:42 UTC] USER=www-data EUID=0 PID=4194157 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[2026-02-05 06:43:42 UTC] USER=www-data EUID=0 PID=4194166 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[INFO] ✅ Backup scripts created
[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-02-05 06:43:42 UTC] USER=www-data EUID=0 PID=4194186 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-web-universe-main-dev
[INFO] ✅ Cron jobs configured
[INFO] Schedule:
[INFO] - Full backup: Sundays at 2:00 AM
[INFO] - Differential backup: Mon-Sat at 2:00 AM
[INFO] 🔟 Creating restore documentation...
[2026-02-05 06:43:42 UTC] USER=www-data EUID=0 PID=4194218 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 06:43:42 UTC] USER=www-data EUID=0 PID=4194236 ACTION=fsop ARGS=sed -i s|__ENV_ID__|web-universe-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 06:43:42 UTC] USER=www-data EUID=0 PID=4194271 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 06:43:42 UTC] USER=www-data EUID=0 PID=4194280 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-02-05 06:43:42.471 P00 INFO: start command begin 2.56.0: --exec-id=300-2bf826ce --log-level-console=info --log-level-file=debug --stanza=web-universe-main-dev-coordinator
2026-02-05 06:43:42.473 P00 WARN: stop file does not exist for stanza web-universe-main-dev-coordinator
2026-02-05 06:43:42.473 P00 INFO: start command end: completed successfully (5ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-02-05 06:43:42.528 P00 INFO: stanza-upgrade command begin 2.56.0: --exec-id=311-3689f976 --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 06:43:42.529 P00 INFO: stanza-upgrade for stanza 'web-universe-main-dev-coordinator' on repo1
2026-02-05 06:43:42.540 P00 INFO: stanza-upgrade command end: completed successfully (15ms)
[INFO] This may take a few minutes depending on database size...
[2026-02-05 06:43:42 UTC] USER=www-data EUID=0 PID=315 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260205-064342.log
[2026-02-05 06:43:42 UTC] USER=www-data EUID=0 PID=326 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260205-064342.log
[2026-02-05 06:43:42 UTC] USER=www-data EUID=0 PID=335 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260205-064342.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-02-05 06:43:51 UTC] USER=www-data EUID=0 PID=605 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-4193460.log /var/log/pgbackrest/initial-backup-20260205-064342.log
[INFO] ✅ Initial full backup completed successfully
[INFO] Log: /var/log/pgbackrest/initial-backup-20260205-064342.log
2026-02-05 06:43:51.406 P00 INFO: repo1: remove expired backup 20260205-054801F
2026-02-05 06:43:51.444 P00 INFO: repo1: 17-3 remove archive, start = 000000010000000000000003, stop = 000000010000000000000005
2026-02-05 06:43:51.445 P00 INFO: repo1: 17-4 no archive to remove
2026-02-05 06:43:51.445 P00 INFO: repo1: 17-5 remove archive, start = 000000010000000000000002, stop = 000000010000000000000002
2026-02-05 06:43:51.445 P00 INFO: expire command end: completed successfully (47ms)
[INFO] Current backups:
stanza: web-universe-main-dev-coordinator
status: ok
cipher: aes-256-cbc
db (prior)
wal archive min/max (17): 000000010000000000000006/000000010000000000000009
full backup: 20260205-054825F
timestamp start/stop: 2026-02-05 05:48:25+00 / 2026-02-05 05:48:33+00
wal start/stop: 000000010000000000000006 / 000000010000000000000006
database size: 33.5MB, database backup size: 33.5MB
repo1: backup set size: 5.4MB, backup size: 5.4MB
db (prior)
wal archive min/max (17): 000000010000000000000004/000000010000000000000008
full backup: 20260205-063608F
timestamp start/stop: 2026-02-05 06:36:08+00 / 2026-02-05 06:36:23+00
wal start/stop: 000000010000000000000004 / 000000010000000000000004
database size: 33.5MB, database backup size: 33.5MB
repo1: backup set size: 5.4MB, backup size: 5.4MB
full backup: 20260205-063637F
timestamp start/stop: 2026-02-05 06:36:37+00 / 2026-02-05 06:36:40+00
wal start/stop: 000000010000000000000007 / 000000010000000000000007
database size: 33.5MB, database backup size: 33.5MB
repo1: backup set size: 5.4MB, backup size: 5.4MB
db (current)
wal archive min/max (17): 000000010000000000000003/000000010000000000000003
full backup: 20260205-064342F
timestamp start/stop: 2026-02-05 06:43:42+00 / 2026-02-05 06:43:51+00
wal start/stop: 000000010000000000000003 / 000000010000000000000003
database size: 33.5MB, database backup size: 33.5MB
repo1: backup set size: 5.4MB, backup size: 5.4MB
[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️ No worker identifier provided - skipping worker backup setup
[INFO] (Run with 'worker-01', 'worker-02', etc. to configure worker backups)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Completed steps:
[INFO] 1. pgBackRest installed and configured
[INFO] 2. WAL archiving enabled (archive_mode=on)
[INFO] 3. PostgreSQL restarted with new settings
[INFO] 4. pgBackRest stanza initialized and verified
[INFO] 5. Initial full backup completed
[INFO] 6. Automated backup cron jobs configured
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO] Coordinator:
[INFO] Stanza: web-universe-main-dev-coordinator
[INFO] Schedule: Full: Sun 2AM, Diff: Mon-Sat 2AM
[INFO] Common:
[INFO] Backup dir: /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO] Archive dir: /var/lib/pgbackrest/archive/web-universe-main-dev
[INFO] Config: /etc/pgbackrest/pgbackrest.conf
[INFO] Restore guide: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] Retention:
[INFO] Full backups: 4 (keep last 4 full backups)
[INFO] Differential: 4 (keep last 4 diff per full)
[INFO] Archive WAL: Auto-managed by pgBackRest
[INFO] Manual commands:
[INFO] Coordinator: sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator backup
[INFO] List all backups: sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up worker backups for 1 worker(s)...
[0;34m[INFO][0m Setting up backup for: worker-01
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for web-universe-main-dev...
[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO] Version: pgBackRest 2.56.0
[INFO] 2️⃣ Creating backup directories...
[2026-02-05 06:43:51 UTC] USER=www-data EUID=0 PID=659 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 06:43:51 UTC] USER=www-data EUID=0 PID=670 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/web-universe-main-dev
[2026-02-05 06:43:51 UTC] USER=www-data EUID=0 PID=679 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-02-05 06:43:51 UTC] USER=www-data EUID=0 PID=691 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-02-05 06:43:52 UTC] USER=www-data EUID=0 PID=700 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-02-05 06:43:52 UTC] USER=www-data EUID=0 PID=711 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-02-05 06:43:53 UTC] USER=www-data EUID=0 PID=792 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-02-05 06:43:53 UTC] USER=www-data EUID=0 PID=801 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-02-05 06:43:53 UTC] USER=www-data EUID=0 PID=812 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-02-05 06:43:53 UTC] USER=www-data EUID=0 PID=821 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/web-universe-main-dev
[2026-02-05 06:43:53 UTC] USER=www-data EUID=0 PID=830 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO] ✅ Backup directories created
[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-web-universe-main-dev
[2026-02-05 06:43:53 UTC] USER=www-data EUID=0 PID=864 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-02-05 06:43:53 UTC] USER=www-data EUID=0 PID=877 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key
[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-02-05 06:43:53 UTC] USER=www-data EUID=0 PID=888 ACTION=fsop ARGS=find /data/postgresql/17/web-universe-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-02-05 06:43:54 UTC] USER=www-data EUID=0 PID=902 ACTION=fsop ARGS=chown -R postgres:postgres /data/postgresql/17/web-universe-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed
[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-02-05 06:43:54 UTC] USER=www-data EUID=0 PID=933 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[INFO] ✅ Spool directory created
[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-02-05 06:43:54 UTC] USER=www-data EUID=0 PID=954 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/web-universe-main-dev/coordinator/PG_VERSION
[2026-02-05 06:43:54 UTC] USER=www-data EUID=0 PID=964 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running
[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza web-universe-main-dev-coordinator already initialized and verified
[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
pg_reload_conf
----------------
t
(1 row)
[INFO] ✅ WAL archiving configured for coordinator
[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-02-05 06:43:54 UTC] USER=www-data EUID=0 PID=1071 ACTION=passthru ARGS=systemctl stop postgresql@web-universe-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-02-05 06:43:57 UTC] USER=www-data EUID=0 PID=1211 ACTION=passthru ARGS=systemctl start postgresql@web-universe-main-dev-coordinator.service
[2026-02-05 06:44:01 UTC] USER=www-data EUID=0 PID=1380 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@web-universe-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled
[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-02-05 06:44:01 UTC] USER=www-data EUID=0 PID=1404 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator --log-level-console=info check
2026-02-05 06:44:01.370 P00 INFO: check command begin 2.56.0: --exec-id=1413-9f9ddc62 --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 06:44:01.389 P00 INFO: check repo1 configuration (primary)
2026-02-05 06:44:01.435 P00 INFO: check repo1 archive for WAL (primary)
2026-02-05 06:44:01.736 P00 INFO: WAL segment 000000010000000000000005 successfully archived to '/var/lib/pgbackrest/backup/web-universe-main-dev/archive/web-universe-main-dev-coordinator/17-5/0000000100000000/000000010000000000000005-03316de15564100034cf22c6c5bc9350102c8764.lz4' on repo1
2026-02-05 06:44:01.736 P00 INFO: check command end: completed successfully (371ms)
[INFO] ✅ Stanza verification passed
[INFO] 8️⃣ Creating backup automation scripts...
[2026-02-05 06:44:01 UTC] USER=www-data EUID=0 PID=1464 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-web-universe-main-dev.sh
[2026-02-05 06:44:01 UTC] USER=www-data EUID=0 PID=1495 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[2026-02-05 06:44:01 UTC] USER=www-data EUID=0 PID=1504 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-web-universe-main-dev.sh
[INFO] ✅ Backup scripts created
[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-02-05 06:44:02 UTC] USER=www-data EUID=0 PID=1523 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-web-universe-main-dev
[INFO] ✅ Cron jobs configured
[INFO] Schedule:
[INFO] - Full backup: Sundays at 2:00 AM
[INFO] - Differential backup: Mon-Sat at 2:00 AM
[INFO] 🔟 Creating restore documentation...
[2026-02-05 06:44:02 UTC] USER=www-data EUID=0 PID=1542 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|web-universe-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 06:44:02 UTC] USER=www-data EUID=0 PID=1567 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/data/postgresql/17/web-universe-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 06:44:02 UTC] USER=www-data EUID=0 PID=1576 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[2026-02-05 06:44:02 UTC] USER=www-data EUID=0 PID=1585 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-02-05 06:44:02.392 P00 INFO: start command begin 2.56.0: --exec-id=1620-404a1201 --log-level-console=info --log-level-file=debug --stanza=web-universe-main-dev-coordinator
2026-02-05 06:44:02.392 P00 WARN: stop file does not exist for stanza web-universe-main-dev-coordinator
2026-02-05 06:44:02.393 P00 INFO: start command end: completed successfully (6ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-02-05 06:44:02.469 P00 INFO: stanza-upgrade command begin 2.56.0: --exec-id=1632-99dc5c01 --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/web-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-web-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/web-universe-main-dev --stanza=web-universe-main-dev-coordinator
2026-02-05 06:44:02.470 P00 INFO: stanza-upgrade for stanza 'web-universe-main-dev-coordinator' on repo1
2026-02-05 06:44:02.471 P00 INFO: stanza 'web-universe-main-dev-coordinator' on repo1 is already up to date
2026-02-05 06:44:02.471 P00 INFO: stanza-upgrade command end: completed successfully (10ms)
[INFO] This may take a few minutes depending on database size...
[2026-02-05 06:44:02 UTC] USER=www-data EUID=0 PID=1636 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260205-064402.log
[2026-02-05 06:44:02 UTC] USER=www-data EUID=0 PID=1645 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260205-064402.log
[2026-02-05 06:44:02 UTC] USER=www-data EUID=0 PID=1656 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260205-064402.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-02-05 06:44:10 UTC] USER=www-data EUID=0 PID=1850 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-630.log /var/log/pgbackrest/initial-backup-20260205-064402.log
[INFO] ✅ Initial full backup completed successfully
[INFO] Log: /var/log/pgbackrest/initial-backup-20260205-064402.log
2026-02-05 06:44:10.594 P00 INFO: repo1: remove expired backup 20260205-054825F
2026-02-05 06:44:10.637 P00 INFO: repo1: remove archive path /var/lib/pgbackrest/backup/web-universe-main-dev/archive/web-universe-main-dev-coordinator/17-3
2026-02-05 06:44:10.643 P00 INFO: repo1: 17-4 no archive to remove
2026-02-05 06:44:10.643 P00 INFO: repo1: 17-5 no archive to remove
2026-02-05 06:44:10.643 P00 INFO: expire command end: completed successfully (58ms)
[INFO] Current backups:
stanza: web-universe-main-dev-coordinator
status: ok
cipher: aes-256-cbc
db (prior)
wal archive min/max (17): 000000010000000000000004/000000010000000000000008
full backup: 20260205-063608F
timestamp start/stop: 2026-02-05 06:36:08+00 / 2026-02-05 06:36:23+00
wal start/stop: 000000010000000000000004 / 000000010000000000000004
database size: 33.5MB, database backup size: 33.5MB
repo1: backup set size: 5.4MB, backup size: 5.4MB
full backup: 20260205-063637F
timestamp start/stop: 2026-02-05 06:36:37+00 / 2026-02-05 06:36:40+00
wal start/stop: 000000010000000000000007 / 000000010000000000000007
database size: 33.5MB, database backup size: 33.5MB
repo1: backup set size: 5.4MB, backup size: 5.4MB
db (current)
wal archive min/max (17): 000000010000000000000003/000000010000000000000005
full backup: 20260205-064342F
timestamp start/stop: 2026-02-05 06:43:42+00 / 2026-02-05 06:43:51+00
wal start/stop: 000000010000000000000003 / 000000010000000000000003
database size: 33.5MB, database backup size: 33.5MB
repo1: backup set size: 5.4MB, backup size: 5.4MB
full backup: 20260205-064402F
timestamp start/stop: 2026-02-05 06:44:02+00 / 2026-02-05 06:44:10+00
wal start/stop: 000000010000000000000006 / 000000010000000000000006
database size: 33.5MB, database backup size: 33.5MB
repo1: backup set size: 5.4MB, backup size: 5.4MB
[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️ No worker identifier provided - skipping worker backup setup
[INFO] (Run with 'worker-01', 'worker-02', etc. to configure worker backups)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Completed steps:
[INFO] 1. pgBackRest installed and configured
[INFO] 2. WAL archiving enabled (archive_mode=on)
[INFO] 3. PostgreSQL restarted with new settings
[INFO] 4. pgBackRest stanza initialized and verified
[INFO] 5. Initial full backup completed
[INFO] 6. Automated backup cron jobs configured
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO] Coordinator:
[INFO] Stanza: web-universe-main-dev-coordinator
[INFO] Schedule: Full: Sun 2AM, Diff: Mon-Sat 2AM
[INFO] Common:
[INFO] Backup dir: /var/lib/pgbackrest/backup/web-universe-main-dev
[INFO] Archive dir: /var/lib/pgbackrest/archive/web-universe-main-dev
[INFO] Config: /etc/pgbackrest/pgbackrest.conf
[INFO] Restore guide: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_web-universe-main-dev.md
[INFO] Retention:
[INFO] Full backups: 4 (keep last 4 full backups)
[INFO] Differential: 4 (keep last 4 diff per full)
[INFO] Archive WAL: Auto-managed by pgBackRest
[INFO] Manual commands:
[INFO] Coordinator: sudo -u postgres pgbackrest --stanza=web-universe-main-dev-coordinator backup
[INFO] List all backups: sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✓[0m ✅ Backup setup completed for coordinator and all workers
[0;34m[INFO][0m Skipping 06-distribute-tables-canary.sh (test script - set RUN_TESTS=true to enable)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 07-distribute-tables.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 06:44:12 UTC] USER=unknown EUID=33 PID=1956 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-02-05 06:44:12 UTC] USER=unknown EUID=33 PID=1963 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-02-05 06:44:12 UTC] USER=unknown EUID=33 PID=1970 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-02-05 06:44:12 UTC] USER=unknown EUID=33 PID=1982 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS TABLE DISTRIBUTION
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m 🔐 Secure connection established
[0;34m[INFO][0m Host: db-web-universe-main-dev-postgresql-coordinator.fastorder.com:5432
[0;34m[INFO][0m Database: fastorder_web_universe_main_dev_db
[0;34m[INFO][0m SSL: verify-full (TLS 1.2+)
[0;34m[INFO][0m Timeouts: statement=120s, idle_tx=300s
[0;34m[INFO][0m 🔍 Running preflight checks...
[0;34m[INFO][0m Testing database connectivity...
[0;32m[OK][0m ✅ Database connection successful
[0;32m[OK][0m ✅ Connected to correct database: fastorder_web_universe_main_dev_db
[0;34m[INFO][0m Checking Citus extension in database fastorder_web_universe_main_dev_db...
[0;32m[OK][0m Citus version: 13.2-1
[0;34m[INFO][0m Checking worker registration...
[0;32m[OK][0m Registered workers: 1
[0;34m[INFO][0m Worker nodes:
[0;34m[INFO][0m nodename | nodeport | isactive | noderole
[0;34m[INFO][0m -------------------------------------------------------------+----------+----------+----------
[0;34m[INFO][0m db-web-universe-main-dev-postgresql-worker-01.fastorder.com | 5432 | t | primary
[0;34m[INFO][0m (1 row)
[0;34m[INFO][0m
[0;34m[INFO][0m 📊 Starting table distribution...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Distributing: auth.login_account
[0;34m[INFO][0m Description: User authentication table - distributed by region for tenant isolation
[0;34m[INFO][0m Shard key: region_hint
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ⏭️ Table does not exist, skipping
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ All tables distributed successfully!
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m 📊 Citus Cluster Summary:
[0;34m[INFO][0m Distributed tables:
[0;34m[INFO][0m table | type | shard_key | shards | size
[0;34m[INFO][0m -------+------+-----------+--------+------
[0;34m[INFO][0m (0 rows)
[0;34m[INFO][0m
[0;34m[INFO][0m Worker capacity:
[0;34m[INFO][0m worker | total_shards | total_size
[0;34m[INFO][0m --------+--------------+------------
[0;34m[INFO][0m (0 rows)
[0;34m[INFO][0m
[0;32m[OK][0m Citus table distribution complete
[0;34m[INFO][0m Skipping 08-distribute-tables-rollback.sh (rollback script - run manually only)
[0;34m[INFO][0m Skipping 09-distribute-tables-test.sh (test script - set RUN_TESTS=true to enable)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 10-setup-cdc.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m CDC PIPELINE SETUP (Debezium + Elasticsearch Sink)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Log file: /var/log/fastorder/cdc/10-setup-cdc-*.log
[0;34m[INFO][0m Running CDC setup for identifier: coordinator
[2026-02-05 06:44:17] ==========================================
[2026-02-05 06:44:17] CDC SETUP SCRIPT STARTED
[2026-02-05 06:44:17] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260205_064417.log
[2026-02-05 06:44:17] ==========================================
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 06:44:18] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 06:44:18] CDC Pipeline Setup (Debezium + ES Sink)
[2026-02-05 06:44:18] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 06:44:18] Environment: web-universe-main-dev
[2026-02-05 06:44:18] Identifier: coordinator
[2026-02-05 06:44:18] Service: web
[2026-02-05 06:44:18] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 06:44:18] 📂 CDC_BASE_DIR exists: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc
[2026-02-05 06:44:18] Looking for service folder: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web
[2026-02-05 06:44:18]
[2026-02-05 06:44:18] 📂 Found CDC configuration for service: web
[2026-02-05 06:44:18] Scanning for subservice directories in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web
[2026-02-05 06:44:18] Found subservice: config, checking for steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps
[2026-02-05 06:44:18]
[2026-02-05 06:44:18] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 06:44:18] Setting up CDC for: web/config
[2026-02-05 06:44:18] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-02-05 06:44:18] Found 3 step script(s) in /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps
[2026-02-05 06:44:18]
[2026-02-05 06:44:18] 🔧 Running: 01-setup-config-cdc.sh
[2026-02-05 06:44:18] Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps/01-setup-config-cdc.sh
[2026-02-05 06:44:18] Executing directly (script is executable)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Config CDC Setup - Automatic Role Detection
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service: web
[INFO] Zone: universe
[INFO] Branch: main
[INFO] Environment: dev
[INFO] Identifier: coordinator
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service 'web' requires config.* schema
[INFO] CDC Role for web in zone universe: master
[INFO] ════════════════════════════════════════════════════════════
[INFO] CONTROL PLANE MASTER (zone=universe)
[INFO] Setting up Debezium CDC Publisher
[INFO] ════════════════════════════════════════════════════════════
[INFO] Executing Debezium config setup...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Debezium Config CDC Master Setup
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: web-universe-main-dev
[INFO] PostgreSQL Host: db-web-universe-main-dev-postgresql.fastorder.com
[INFO] Database: fastorder_web_universe_main_dev_db
[INFO] Connector: debezium-web-universe-main-dev-config
[INFO] Kafka Connect: eventbus-web-universe-main-dev-kafka-connect.fastorder.com:8083
[INFO] SSL Cert Dir: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator (on Kafka Connect host)
[INFO] SSL Key File: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der
[INFO] Topic Prefix: cdc.web_universe_main_dev
[INFO] Replication Slot: dbz_web_universe_main_dev_config
[INFO] Tables: config.public_defaults,config.feature_flags,config.config_version
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Step 1: Creating PostgreSQL publication for config tables...
[INFO] Creating publication: cdc_pub_web_config
CREATE PUBLICATION
[INFO] Setting REPLICA IDENTITY FULL for config tables...
ALTER TABLE
ALTER TABLE
ALTER TABLE
[INFO] Step 2: Creating replication slot...
[INFO] Creating replication slot: dbz_web_universe_main_dev_config
(dbz_web_universe_main_dev_config,0/700E210)
[INFO] Step 3: Registering Debezium connector with Kafka Connect...
[INFO] Creating new connector: debezium-web-universe-main-dev-config
[INFO] Sending connector configuration to Kafka Connect...
[ERROR] Failed to register Debezium connector
[ERROR] Response: {"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nError while validating connector config: FATAL: password authentication failed for user \"debezium_user\"\nYou can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`"}
[ OK ] Debezium config CDC master setup complete
[INFO] No topology.json found at /opt/fastorder/state/web-universe-main-dev/topology.json - skipping merge
[INFO]
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Config CDC Setup Complete
[INFO] ═══════════════════════════════════════════════════════════════
═══════════════════════════════════════════════════════════════
Service Capabilities: web-universe-main-dev
═══════════════════════════════════════════════════════════════
Service: web
Zone: universe
Branch: main
Environment: dev
Config Schema: ✅ YES
Redis Cache: ✅ YES
CDC Role: master
CDC Master Configuration:
Debezium: debezium-web-universe-main-dev-config
Topic Prefix: cdc.web_universe_main_dev
Repl Slot: dbz_web_universe_main_dev_config
Tables: config.public_defaults,config.feature_flags,config.config_version
Required Schemas: config tenant dashboard environment resource service item company communication ai
═══════════════════════════════════════════════════════════════
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]
[INFO] Log file: /var/log/fastorder/cdc/config-cdc-20260205_064418.log
[ OK ] Config CDC setup finished successfully
[2026-02-05 06:44:43] ✅ Completed: 01-setup-config-cdc.sh
[2026-02-05 06:44:43]
[2026-02-05 06:44:43] 🔧 Running: 02-setup-debezium-config.sh
[2026-02-05 06:44:43] Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/web/config/steps/02-setup-debezium-config.sh
[2026-02-05 06:44:43] Executing directly (script is executable)
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Debezium Config CDC Master Setup
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: web-universe-main-dev
[INFO] PostgreSQL Host: db-web-universe-main-dev-postgresql.fastorder.com
[INFO] Database: fastorder_web_universe_main_dev_db
[INFO] Connector: debezium-web-universe-main-dev-config
[INFO] Kafka Connect: eventbus-web-universe-main-dev-kafka-connect.fastorder.com:8083
[INFO] SSL Cert Dir: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator (on Kafka Connect host)
[INFO] SSL Key File: /etc/fastorder/postgresql/certs/web-universe-main-dev/coordinator/debezium_user_pk8.der
[INFO] Topic Prefix: cdc.web_universe_main_dev
[INFO] Replication Slot: dbz_web_universe_main_dev_config
[INFO] Tables: config.public_defaults,config.feature_flags,config.config_version
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Step 1: Creating PostgreSQL publication for config tables...
[INFO] Publication cdc_pub_web_config already exists
[INFO] Setting REPLICA IDENTITY FULL for config tables...
ALTER TABLE
ALTER TABLE
ALTER TABLE
[INFO] Step 2: Creating replication slot...
[INFO] Replication slot dbz_web_universe_main_dev_config already exists
[INFO] Step 3: Registering Debezium connector with Kafka Connect...
[INFO] Creating new connector: debezium-web-universe-main-dev-config
[INFO] Sending connector configuration to Kafka Connect...
[ERROR] Failed to register Debezium connector
[ERROR] Response: {"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nError while validating connector config: FATAL: password authentication failed for user \"debezium_user\"\nYou can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`"}
[2026-02-05 06:45:12] ❌ FAILED: 02-setup-debezium-config.sh (exit code: 1)
[2026-02-05 06:45:12] ❌ CRITICAL: This is a required step for CDC pipeline. Aborting.
[0;31m[ERROR][0m ❌ Database infrastructure (postgresql) setup failed with exit code: 1
⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...