📊 Provisioning Job Status

[INFO] Using eventbus engine from EVENTBUS_ENGINE environment variable: kafka
[INFO] Cleaning up any existing locks...

Starting eventbus engine: kafka
═══════════════════════════════════════════════

[INFO] Using environment from web interface: zone-universe-main-dev
[2026-02-05 09:57:31] Using web-provided environment: zone-universe-main-dev
[2026-02-05 09:57:31] Service: zone, Zone: universe, Branch: main, Env: dev
✓ Environment initialized successfully (mode: general)
[INFO] Starting Kafka setup process...
[INFO] Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps
[INFO] Environment: zone-universe-main-dev

[INFO] Found 13 step(s) to execute

[INFO] 📦 Step 1/13: install debezium connector...
═══════════════════════════════════════════════════════════════════
Fetching latest versions from Maven Central...
Installing Debezium PostgreSQL Connector
  Debezium version: 3.4.1.Final
  pgjdbc version:   42.7.9
═══════════════════════════════════════════════════════════════════
[OK] Debezium 3.4.1.Final with pgjdbc 42.7.9 already installed
[OK] ✅ Step 1 completed: 00-install-debezium-connector.sh

[INFO] 📦 Step 2/13: kafka setup...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials for secrets vault...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
🧹 Checking for orphaned Kafka processes on ports 9092, 9093, 8083...
  ⚠️  Found process on port 9092 (PIDs: [2026-02-05 09:57:32 UTC] USER=www-data EUID=0 PID=632297 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 2>/dev/null || true), killing...
[2026-02-05 09:57:32 UTC] USER=www-data EUID=0 PID=632319 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 09:57:32 UTC] USER=www-data EUID=0 PID=632297 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 2>/dev/null || true 2>/dev/null || true
  ⚠️  Found process on port 9093 (PIDs: [2026-02-05 09:57:33 UTC] USER=www-data EUID=0 PID=632374 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 2>/dev/null || true
36080
36233
36235
48970
580708
588343
592278
592356
592399
592401
601981
602049
603839), killing...
[2026-02-05 09:57:34 UTC] USER=www-data EUID=0 PID=632399 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 09:57:33 UTC] USER=www-data EUID=0 PID=632374 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 2>/dev/null || true
36080
36233
36235
48970
580708
588343
592278
592356
592399
592401
601981
602049
603839 2>/dev/null || true
/usr/bin/bash: line 2: 36080: command not found
/usr/bin/bash: line 3: 36233: command not found
/usr/bin/bash: line 4: 36235: command not found
/usr/bin/bash: line 5: 48970: command not found
/usr/bin/bash: line 6: 580708: command not found
/usr/bin/bash: line 7: 588343: command not found
/usr/bin/bash: line 8: 592278: command not found
/usr/bin/bash: line 9: 592356: command not found
/usr/bin/bash: line 10: 592399: command not found
/usr/bin/bash: line 11: 592401: command not found
/usr/bin/bash: line 12: 601981: command not found
/usr/bin/bash: line 13: 602049: command not found
  ⚠️  Found process on port 8083 (PIDs: [2026-02-05 09:57:35 UTC] USER=www-data EUID=0 PID=632484 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 2>/dev/null || true
35054
36234
36236), killing...
[2026-02-05 09:57:35 UTC] USER=www-data EUID=0 PID=632494 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 09:57:35 UTC] USER=www-data EUID=0 PID=632484 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 2>/dev/null || true
35054
36234
36236 2>/dev/null || true
/usr/bin/bash: line 2: 35054: command not found
/usr/bin/bash: line 3: 36234: command not found
✅ Port cleanup completed
Ensuring KAFKA application environment for coordinator...
[INFO] Using existing KAFKA environment: eventbus-zone-universe-main-dev-kafka-connect.fastorder.com (10.100.1.110)
Ensuring KAFKA_BROKER_IP application environment for coordinator...
[INFO] Using existing KAFKA BROKER environment: eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com (10.100.1.52)
[INFO] Kafka Broker IP: 10.100.1.52
[INFO] Kafka Connect IP: 10.100.1.110
[INFO] Registered /etc/hosts: eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com -> 10.100.1.52
[INFO] Registered /etc/hosts: eventbus-zone-universe-main-dev-kafka-connect.fastorder.com -> 10.100.1.110
🔐 Initializing keystore passwords...
[INFO] 🔍 Checking secrets backend (provider: aws)...
⚠️  No existing passwords found
[INFO] 🔐 Generating new random passwords...
✅ Generated new passwords
[INFO] 🔐 Vaulting kafka passwords to remote backend...
✅ Passwords vaulted to remote backend
✅ Keystore passwords initialized
   - Keystore password: sfDzrJOG... (32 chars)
   - Truststore password: JBH3gyVt... (32 chars)
[INFO] 🔐 Vaulting kafka passwords to remote backend...
✅ Passwords vaulted to remote backend
✅ Kafka keystore passwords saved to AWS Secrets Manager
[INFO] Generating for: zone-universe-main-dev (host=eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com ip=10.100.1.52)
[2026-02-05 09:57:48 UTC] USER=www-data EUID=0 PID=632959 ACTION=fsop ARGS=rm -rf /opt/kafka/secrets/zone-universe-main-dev/coordinator /data/kafka/zone-universe-main-dev/coordinator
[2026-02-05 09:57:49 UTC] USER=www-data EUID=0 PID=632979 ACTION=fsop ARGS=mkdir -p /opt/kafka/secrets/zone-universe-main-dev/coordinator /opt/kafka/config/zone-universe-main-dev/coordinator /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem /data/kafka/zone-universe-main-dev_coordinator-data
[2026-02-05 09:57:49 UTC] USER=www-data EUID=0 PID=632993 ACTION=fsop ARGS=chown -R kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator
[2026-02-05 09:57:49 UTC] USER=www-data EUID=0 PID=633003 ACTION=fsop ARGS=chown -R kafka:kafka /opt/kafka/config/zone-universe-main-dev/coordinator /data/kafka/zone-universe-main-dev_coordinator-data
[2026-02-05 09:57:49 UTC] USER=www-data EUID=0 PID=633012 ACTION=fsop ARGS=chmod 770 /opt/kafka/config/zone-universe-main-dev/coordinator /data/kafka/zone-universe-main-dev_coordinator-data
[2026-02-05 09:57:49 UTC] USER=www-data EUID=0 PID=633021 ACTION=fsop ARGS=chmod 750 /opt/kafka/secrets/zone-universe-main-dev/coordinator
[2026-02-05 09:57:49 UTC] USER=www-data EUID=0 PID=633030 ACTION=fsop ARGS=chmod 750 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem
[2026-02-05 09:57:49 UTC] USER=www-data EUID=0 PID=633040 ACTION=fsop ARGS=chmod 700 /tmp/fo-tls.q9sgdB
[2026-02-05 09:57:49 UTC] USER=www-data EUID=0 PID=633049 ACTION=fsop ARGS=chmod 755 /tmp/fo-tls.q9sgdB
[2026-02-05 09:57:49 UTC] USER=www-data EUID=0 PID=633058 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/fo-tls.q9sgdB/ra_root.crt
[2026-02-05 09:57:49 UTC] USER=www-data EUID=0 PID=633067 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/fo-tls.q9sgdB/ra_root.key
[2026-02-05 09:57:49 UTC] USER=www-data EUID=0 PID=633086 ACTION=fsop ARGS=chmod 644 /tmp/fo-tls.q9sgdB/ra_root.key
Certificate was added to keystore
[2026-02-05 09:57:50 UTC] USER=www-data EUID=0 PID=633169 ACTION=fsop ARGS=mv /tmp/fo-tls.q9sgdB/truststore.jks /opt/kafka/secrets/zone-universe-main-dev/coordinator/truststore.jks
[2026-02-05 09:57:50 UTC] USER=www-data EUID=0 PID=633179 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/zone-universe-main-dev/coordinator/truststore.jks
Generating 4,096 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 825 days
	for: CN=eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com, OU=Kafka Broker, O=FastOrder, C=AE

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /tmp/fo-tls.q9sgdB/kafka.server.keystore.jks -destkeystore /tmp/fo-tls.q9sgdB/kafka.server.keystore.jks -deststoretype pkcs12".

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /tmp/fo-tls.q9sgdB/kafka.server.keystore.jks -destkeystore /tmp/fo-tls.q9sgdB/kafka.server.keystore.jks -deststoretype pkcs12".
Certificate request self-signature ok
subject=C = AE, O = FastOrder, OU = Kafka Broker, CN = eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com
Certificate was added to keystore

Warning:
Certificate reply was installed in keystore

Warning:
[2026-02-05 09:57:56 UTC] USER=www-data EUID=0 PID=633537 ACTION=fsop ARGS=mv /tmp/fo-tls.q9sgdB/kafka.server.keystore.jks /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 09:57:57 UTC] USER=www-data EUID=0 PID=633546 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 09:57:57 UTC] USER=www-data EUID=0 PID=633555 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.server.keystore.jks
Generating 4,096 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 825 days
	for: CN=eventbus-zone-universe-main-dev-kafka-connect.fastorder.com, OU=Kafka Connect REST, O=FastOrder, C=AE
Certificate request self-signature ok
subject=C = AE, O = FastOrder, OU = Kafka Connect REST, CN = eventbus-zone-universe-main-dev-kafka-connect.fastorder.com
Certificate was added to keystore
Certificate reply was installed in keystore
[2026-02-05 09:58:01 UTC] USER=www-data EUID=0 PID=633787 ACTION=fsop ARGS=mv /tmp/fo-tls.q9sgdB/connect-rest.keystore.p12 /opt/kafka/secrets/zone-universe-main-dev/coordinator/connect-rest.keystore.p12
[2026-02-05 09:58:01 UTC] USER=www-data EUID=0 PID=633796 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/zone-universe-main-dev/coordinator/connect-rest.keystore.p12
[2026-02-05 09:58:01 UTC] USER=www-data EUID=0 PID=633809 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/connect-rest.keystore.p12
Certificate request self-signature ok
subject=CN = kafka-client-zone-universe-main-dev, OU = Kafka Client, O = FastOrder, C = AE
[2026-02-05 09:58:02 UTC] USER=www-data EUID=0 PID=633851 ACTION=fsop ARGS=cp /tmp/fo-tls.q9sgdB/ra_root.crt /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem
[2026-02-05 09:58:02 UTC] USER=www-data EUID=0 PID=633860 ACTION=fsop ARGS=cp /tmp/fo-tls.q9sgdB/client-key.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 09:58:02 UTC] USER=www-data EUID=0 PID=633870 ACTION=fsop ARGS=cp /tmp/fo-tls.q9sgdB/client-cert.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
[2026-02-05 09:58:02 UTC] USER=www-data EUID=0 PID=633880 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-cert.pem
[2026-02-05 09:58:02 UTC] USER=www-data EUID=0 PID=633899 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 09:58:02 UTC] USER=www-data EUID=0 PID=633910 ACTION=fsop ARGS=mv /tmp/fo-tls.q9sgdB/kafka.client.keystore.p12 /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.client.keystore.p12
[2026-02-05 09:58:02 UTC] USER=www-data EUID=0 PID=633919 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.client.keystore.p12
[2026-02-05 09:58:02 UTC] USER=www-data EUID=0 PID=633928 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.client.keystore.p12
🔐 Ensuring kafka user has access to PostgreSQL certificates...
✅ kafka is already in postgres group
🧹 Cleaning up conflicting services and processes on Kafka ports on 10.100.1.52...
🔪 Killing processes on 10.100.1.52:8083: [2026-02-05 09:58:02 UTC] USER=www-data EUID=0 PID=633976 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.52:8083 -t 2>/dev/null || true
[2026-02-05 09:58:03 UTC] USER=www-data EUID=0 PID=634001 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 09:58:02 UTC] USER=www-data EUID=0 PID=633976 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.52:8083 -t 2>/dev/null || true
🔪 Killing processes on 10.100.1.52:9092: [2026-02-05 09:58:03 UTC] USER=www-data EUID=0 PID=634019 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.52:9092 -t 2>/dev/null || true
[2026-02-05 09:58:03 UTC] USER=www-data EUID=0 PID=634039 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 09:58:03 UTC] USER=www-data EUID=0 PID=634019 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.52:9092 -t 2>/dev/null || true
🔪 Killing processes on 10.100.1.52:9093: [2026-02-05 09:58:03 UTC] USER=www-data EUID=0 PID=634050 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.52:9093 -t 2>/dev/null || true
[2026-02-05 09:58:04 UTC] USER=www-data EUID=0 PID=634129 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 09:58:03 UTC] USER=www-data EUID=0 PID=634050 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.52:9093 -t 2>/dev/null || true
✅ Port cleanup completed
🔧 Checking for Kafka Connect internal topics with incorrect cleanup policy...
ℹ️  Kafka broker not running, skipping topic cleanup policy fix
🔧 Creating environment-specific systemd units...
🔧 Writing client properties to /etc/kafka/client-zone-universe-main-dev-coordinator.properties ...
[2026-02-05 09:58:04 UTC] USER=www-data EUID=0 PID=634168 ACTION=fsop ARGS=chown root:kafka /etc/kafka/client-zone-universe-main-dev-coordinator.properties
[2026-02-05 09:58:04 UTC] USER=www-data EUID=0 PID=634177 ACTION=fsop ARGS=chmod 0644 /etc/kafka/client-zone-universe-main-dev-coordinator.properties
🔧 Creating PEM certificates for PHP mTLS access...
[2026-02-05 09:58:04 UTC] USER=www-data EUID=0 PID=634187 ACTION=passthru ARGS=bash -c openssl pkcs12 -in '/opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.client.keystore.p12' -clcerts -nokeys -passin pass:'sfDzrJOGShZ7iICxdMeMU72wUc1Es4tN' -out '/opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client.crt' 2>/dev/null
[2026-02-05 09:58:04 UTC] USER=www-data EUID=0 PID=634198 ACTION=passthru ARGS=bash -c openssl pkcs12 -in '/opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.client.keystore.p12' -nocerts -nodes -passin pass:'sfDzrJOGShZ7iICxdMeMU72wUc1Es4tN' -out '/opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client.key' 2>/dev/null
[2026-02-05 09:58:04 UTC] USER=www-data EUID=0 PID=634208 ACTION=passthru ARGS=bash -c keytool -exportcert -alias fastorder-ra-root -keystore '/opt/kafka/secrets/zone-universe-main-dev/coordinator/truststore.jks' -storepass 'JBH3gyVtYBjR06bIlHqf0792HMBfdcRK' -rfc -file '/opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.crt' 2>/dev/null
[2026-02-05 09:58:05 UTC] USER=www-data EUID=0 PID=634255 ACTION=fsop ARGS=chown root:kafka /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client.crt /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client.key /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.crt
[2026-02-05 09:58:05 UTC] USER=www-data EUID=0 PID=634264 ACTION=fsop ARGS=chmod 0644 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client.crt /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/ca.crt
[2026-02-05 09:58:05 UTC] USER=www-data EUID=0 PID=634273 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/pem/client.key
🔐 Creating connector secrets file for FileConfigProvider...
[2026-02-05 09:58:05 UTC] USER=www-data EUID=0 PID=634335 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/zone-universe-main-dev/coordinator/connector-secrets.properties
[2026-02-05 09:58:05 UTC] USER=www-data EUID=0 PID=634344 ACTION=fsop ARGS=chmod 0600 /opt/kafka/secrets/zone-universe-main-dev/coordinator/connector-secrets.properties
✅ Connector secrets file created: /opt/kafka/secrets/zone-universe-main-dev/coordinator/connector-secrets.properties
   FileConfigProvider syntax: ${file:/opt/kafka/secrets/zone-universe-main-dev/coordinator/connector-secrets.properties:key_name}
🔧 Creating Canary Event timer for pipeline verification...
[2026-02-05 09:58:05 UTC] USER=www-data EUID=0 PID=634371 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 09:58:06 UTC] USER=www-data EUID=0 PID=634434 ACTION=passthru ARGS=systemctl enable kafka-canary-zone-universe-main-dev.timer
Created symlink /etc/systemd/system/timers.target.wants/kafka-canary-zone-universe-main-dev.timer → /etc/systemd/system/kafka-canary-zone-universe-main-dev.timer.
[2026-02-05 09:58:07 UTC] USER=www-data EUID=0 PID=634500 ACTION=passthru ARGS=systemctl start kafka-canary-zone-universe-main-dev.timer
✅ Canary timer installed: kafka-canary-zone-universe-main-dev.timer (every 5 minutes)
[2026-02-05 09:58:07 UTC] USER=www-data EUID=0 PID=634510 ACTION=passthru ARGS=systemctl daemon-reload
Failed to print table: Broken pipe
Failed to print table: Broken pipe
🔒 Adjusting group ownership and permissions ...
[2026-02-05 09:58:16 UTC] USER=www-data EUID=0 PID=634664 ACTION=fsop ARGS=chown :kafka /opt/kafka/secrets/zone-universe-main-dev/coordinator/truststore.jks /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 09:58:16 UTC] USER=www-data EUID=0 PID=634700 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/truststore.jks /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 09:58:16 UTC] USER=www-data EUID=0 PID=634790 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.client.keystore.p12
[2026-02-05 09:58:16 UTC] USER=www-data EUID=0 PID=634827 ACTION=fsop ARGS=chown root:kafka /etc/kafka/client-zone-universe-main-dev-coordinator.properties
[2026-02-05 09:58:16 UTC] USER=www-data EUID=0 PID=634846 ACTION=fsop ARGS=chmod 0644 /etc/kafka/client-zone-universe-main-dev-coordinator.properties

✅ Kafka configuration complete for zone-universe-main-dev_coordinator
  Broker ID         : 57
  Broker keystore   : /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.server.keystore.jks
  REST keystore     : /opt/kafka/secrets/zone-universe-main-dev/coordinator/connect-rest.keystore.p12
  Truststore        : /opt/kafka/secrets/zone-universe-main-dev/coordinator/truststore.jks
  Client PKCS12     : /opt/kafka/secrets/zone-universe-main-dev/coordinator/kafka.client.keystore.p12
  Data directory    : /data/kafka/zone-universe-main-dev_coordinator-data
  Server config     : /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
  Connect config    : /opt/kafka/config/zone-universe-main-dev/coordinator/connect-distributed.properties
  CLI client config : /etc/kafka/client-zone-universe-main-dev-coordinator.properties

🎯 Next step: Run 03-restart-kafka-related-services.sh to start services

[OK] ✅ Step 2 completed: 01-kafka-setup.sh

[INFO] 📦 Step 3/13: metadata...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] 🎯 Kafka metadata mode: kraft

╔════════════════════════════════════════════════════════════════════╗
║            Kafka Metadata Layer Setup                             ║
╚════════════════════════════════════════════════════════════════════╝

  Environment    : zone-universe-main-dev
  Service        : zone
  Zone           : universe
  Branch         : main
  Environment    : dev
  VM IP          : 142.93.238.16
  Metadata Mode  : kraft

  📋 KRaft Mode (Modern)
  ────────────────────────────────────────────────────────────────
  ✅ No ZooKeeper dependency
  ✅ Faster metadata operations
  ✅ Simplified architecture
  ✅ Recommended for new deployments
  ⚠️  Requires Kafka 3.3+ in production


════════════════════════════════════════════════════════════════════

[INFO] 🚀 Executing KRaft setup script...
[INFO] Script: /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps/metadata/kraft.sh

[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[2026-02-05 09:58:17 UTC] USER=www-data EUID=0 PID=634964 ACTION=fsop ARGS=mkdir -p /data/kafka/zone-universe-main-dev_coordinator-meta /opt/kafka/config/zone-universe-main-dev/coordinator /data/kafka/zone-universe-main-dev_coordinator-data
[2026-02-05 09:58:17 UTC] USER=www-data EUID=0 PID=634973 ACTION=fsop ARGS=chown -R kafka:kafka /data/kafka/zone-universe-main-dev_coordinator-meta /opt/kafka/config/zone-universe-main-dev/coordinator /data/kafka/zone-universe-main-dev_coordinator-data
[2026-02-05 09:58:17 UTC] USER=www-data EUID=0 PID=634982 ACTION=fsop ARGS=chmod 770 /data/kafka/zone-universe-main-dev_coordinator-meta /opt/kafka/config/zone-universe-main-dev/coordinator /data/kafka/zone-universe-main-dev_coordinator-data
[INFO] Adding eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com to /etc/hosts -> 10.100.1.52
[INFO] Adding eventbus-zone-universe-main-dev-kafka-connect.fastorder.com to /etc/hosts -> 10.100.1.110
[INFO] Setting up KRaft for: zone-universe-main-dev (host=eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com ip=10.100.1.52)
[2026-02-05 09:58:17 UTC] USER=www-data EUID=0 PID=635006 ACTION=fsop ARGS=mkdir -p /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev
[INFO] Generated cluster.id=8Ts5q7jcRsa8jOqSp19b1g

🔧 Configuring Kafka for KRaft mode...
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635568 ACTION=fsop ARGS=test -f /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635577 ACTION=fsop ARGS=test -r /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635586 ACTION=fsop ARGS=sed -i /^zookeeper\.connect=/d /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635595 ACTION=passthru ARGS=bash -c grep -q '^process.roles=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635622 ACTION=passthru ARGS=bash -c grep -q '^node.id=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635665 ACTION=passthru ARGS=bash -c grep -q '^broker.id=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635674 ACTION=fsop ARGS=sed -i s|^broker.id=.*|broker.id=1| /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635683 ACTION=passthru ARGS=bash -c grep -q '^controller.listener.names=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635702 ACTION=passthru ARGS=bash -c grep -q '^controller.quorum.voters=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635721 ACTION=passthru ARGS=bash -c grep -q '^metadata.log.dir=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635740 ACTION=passthru ARGS=bash -c grep -q '^log.dirs=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635749 ACTION=fsop ARGS=sed -i s|^log.dirs=.*|log.dirs=/data/kafka/zone-universe-main-dev_coordinator-data| /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635758 ACTION=passthru ARGS=bash -c grep -q '^listeners=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635767 ACTION=passthru ARGS=bash -c grep -q 'CONTROLLER://' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635776 ACTION=fsop ARGS=sed -i s|^listeners=.*|listeners=SSL://10.100.1.52:9092,CONTROLLER://10.100.1.52:9093| /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635786 ACTION=passthru ARGS=bash -c grep -q '^advertised.listeners=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635795 ACTION=fsop ARGS=sed -i s|^advertised.listeners=.*|advertised.listeners=SSL://eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com:9092| /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635804 ACTION=passthru ARGS=bash -c grep -q '^listener.security.protocol.map=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635813 ACTION=fsop ARGS=sed -i s|^listener.security.protocol.map=.*|listener.security.protocol.map=SSL:SSL,CONTROLLER:PLAINTEXT| /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635822 ACTION=passthru ARGS=bash -c grep -q '^inter.broker.listener.name=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635832 ACTION=fsop ARGS=sed -i s|^inter.broker.listener.name=.*|inter.broker.listener.name=SSL| /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635841 ACTION=passthru ARGS=bash -c grep -q '^offsets.topic.replication.factor=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 09:58:22 UTC] USER=www-data EUID=0 PID=635850 ACTION=fsop ARGS=sed -i s|^offsets.topic.replication.factor=.*|offsets.topic.replication.factor=1| /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 09:58:22 UTC] USER=www-data EUID=0 PID=635859 ACTION=passthru ARGS=bash -c grep -q '^transaction.state.log.replication.factor=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 09:58:22 UTC] USER=www-data EUID=0 PID=635868 ACTION=fsop ARGS=sed -i s|^transaction.state.log.replication.factor=.*|transaction.state.log.replication.factor=1| /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 09:58:22 UTC] USER=www-data EUID=0 PID=635877 ACTION=passthru ARGS=bash -c grep -q '^transaction.state.log.min.isr=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
[2026-02-05 09:58:22 UTC] USER=www-data EUID=0 PID=635886 ACTION=fsop ARGS=sed -i s|^transaction.state.log.min.isr=.*|transaction.state.log.min.isr=1| /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties
[2026-02-05 09:58:22 UTC] USER=www-data EUID=0 PID=635897 ACTION=passthru ARGS=bash -c grep -q '^min.insync.replicas=' '/opt/kafka/config/zone-universe-main-dev/coordinator/server.properties'
  ✅ KRaft configuration applied to server.properties
[2026-02-05 09:58:22 UTC] USER=www-data EUID=0 PID=635916 ACTION=fsop ARGS=test -f /data/kafka/zone-universe-main-dev_coordinator-meta/meta.properties
[INFO] Formatting storage directories for coordinator (cluster.id=[2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635559 ACTION=passthru ARGS=bash -c cat /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/kafka_kraft_cluster_id)
[2026-02-05 09:58:22 UTC] USER=www-data EUID=0 PID=635925 ACTION=passthru ARGS=sudo -u kafka /opt/kafka/bin/kafka-storage.sh format --config /opt/kafka/config/zone-universe-main-dev/coordinator/server.properties --cluster-id [2026-02-05 09:58:21 UTC] USER=www-data EUID=0 PID=635559 ACTION=passthru ARGS=bash -c cat /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/kafka_kraft_cluster_id --ignore-formatted
Formatting metadata directory /data/kafka/zone-universe-main-dev_coordinator-meta with metadata.version 3.9-IV0.
Formatting data directory /data/kafka/zone-universe-main-dev_coordinator-data with metadata.version 3.9-IV0.
[INFO] Formatted: /data/kafka/zone-universe-main-dev_coordinator-meta and /data/kafka/zone-universe-main-dev_coordinator-data
🔧 Creating/refreshing KRaft systemd unit...
[2026-02-05 09:58:25 UTC] USER=www-data EUID=0 PID=636366 ACTION=fsop ARGS=sed -i s|\\$MAINPID|$MAINPID|g /etc/systemd/system/confluent-kraft-zone-universe-main-dev_coordinator.service
[2026-02-05 09:58:25 UTC] USER=www-data EUID=0 PID=636375 ACTION=passthru ARGS=systemctl daemon-reload
  ✅ Ensured confluent-kraft-zone-universe-main-dev_coordinator.service
🛑 Stopping legacy ZooKeeper-mode services and current KRaft instance...
  🧹 Cleaning up rogue Kafka processes...
  🧹 Killing any processes holding Kafka ports 9092, 9093...
  🔪 Killing processes on port 9092: 35054
36233
36234
36235
36236
40299
40996
48970
[2026-02-05 09:58:26 UTC] USER=www-data EUID=0 PID=636483 ACTION=passthru ARGS=bash -c kill -9 35054
[2026-02-05 09:58:26 UTC] USER=www-data EUID=0 PID=636540 ACTION=passthru ARGS=bash -c kill -9 36233
[2026-02-05 09:58:26 UTC] USER=www-data EUID=0 PID=636582 ACTION=passthru ARGS=bash -c kill -9 36235
[2026-02-05 09:58:26 UTC] USER=www-data EUID=0 PID=636607 ACTION=passthru ARGS=bash -c kill -9 36236
[2026-02-05 09:58:26 UTC] USER=www-data EUID=0 PID=636620 ACTION=passthru ARGS=bash -c kill -9 40299
[2026-02-05 09:58:26 UTC] USER=www-data EUID=0 PID=636632 ACTION=passthru ARGS=bash -c kill -9 40996
[2026-02-05 09:58:27 UTC] USER=www-data EUID=0 PID=636648 ACTION=passthru ARGS=bash -c kill -9 48970
  🔪 Killing processes on port 9093: 36080
580708
588343
592278
592356
592399
592401
601981
602049
603839
[2026-02-05 09:58:29 UTC] USER=www-data EUID=0 PID=636822 ACTION=passthru ARGS=bash -c kill -9 36080
[2026-02-05 09:58:29 UTC] USER=www-data EUID=0 PID=636844 ACTION=passthru ARGS=bash -c kill -9 588343
[2026-02-05 09:58:29 UTC] USER=www-data EUID=0 PID=636862 ACTION=passthru ARGS=bash -c kill -9 592356
[2026-02-05 09:58:29 UTC] USER=www-data EUID=0 PID=636871 ACTION=passthru ARGS=bash -c kill -9 592399
[2026-02-05 09:58:29 UTC] USER=www-data EUID=0 PID=636880 ACTION=passthru ARGS=bash -c kill -9 592401
[2026-02-05 09:58:29 UTC] USER=www-data EUID=0 PID=636889 ACTION=passthru ARGS=bash -c kill -9 601981
[2026-02-05 09:58:29 UTC] USER=www-data EUID=0 PID=636907 ACTION=passthru ARGS=bash -c kill -9 603839
  ✅ Legacy services stopped and rogue processes cleaned
🔓 Removing stale lock files...
[2026-02-05 09:58:35 UTC] USER=www-data EUID=0 PID=637106 ACTION=fsop ARGS=test -f /data/kafka/zone-universe-main-dev_coordinator-meta/.lock
[2026-02-05 09:58:35 UTC] USER=www-data EUID=0 PID=637115 ACTION=fsop ARGS=test -f /data/kafka/zone-universe-main-dev_coordinator-data/.lock
  ✅ Lock file check complete
🚀 Starting confluent-kraft-zone-universe-main-dev_coordinator.service ...
[2026-02-05 09:58:35 UTC] USER=www-data EUID=0 PID=637124 ACTION=passthru ARGS=systemctl enable confluent-kraft-zone-universe-main-dev_coordinator.service
Created symlink /etc/systemd/system/multi-user.target.wants/confluent-kraft-zone-universe-main-dev_coordinator.service → /etc/systemd/system/confluent-kraft-zone-universe-main-dev_coordinator.service.
[2026-02-05 09:58:36 UTC] USER=www-data EUID=0 PID=637171 ACTION=passthru ARGS=systemctl restart confluent-kraft-zone-universe-main-dev_coordinator.service
🔧 Patching shared Connect unit to follow KRaft broker...
[2026-02-05 09:58:39 UTC] USER=www-data EUID=0 PID=637780 ACTION=fsop ARGS=sed -i -e s|${FULL_ENV}|zone-universe-main-dev|g -e s|${IDENTIFIER}|coordinator|g -e s|${CONFIG_DIR}|/opt/kafka/config/zone-universe-main-dev/coordinator|g /etc/systemd/system/confluent-connect-zone-universe-main-dev_coordinator.service
[2026-02-05 09:58:39 UTC] USER=www-data EUID=0 PID=637791 ACTION=fsop ARGS=sed -i s|\\$MAINPID|$MAINPID|g /etc/systemd/system/confluent-connect-zone-universe-main-dev_coordinator.service
[2026-02-05 09:58:39 UTC] USER=www-data EUID=0 PID=637803 ACTION=fsop ARGS=sed -i s|^After=.*|After=network-online.target confluent-kraft-zone-universe-main-dev_coordinator.service| /etc/systemd/system/confluent-connect-zone-universe-main-dev_coordinator.service
[2026-02-05 09:58:39 UTC] USER=www-data EUID=0 PID=637826 ACTION=fsop ARGS=sed -i s|^ExecStart=.*|ExecStart=/opt/kafka/bin/connect-distributed.sh /opt/kafka/config/zone-universe-main-dev/coordinator/connect-distributed.properties| /etc/systemd/system/confluent-connect-zone-universe-main-dev_coordinator.service
[2026-02-05 09:58:39 UTC] USER=www-data EUID=0 PID=637841 ACTION=passthru ARGS=systemctl daemon-reload
  ✅ Connect unit patched
[2026-02-05 09:58:40 UTC] USER=www-data EUID=0 PID=637915 ACTION=fsop ARGS=ln -sf /opt/kafka/config/zone-universe-main-dev/coordinator/connect-distributed.properties /opt/kafka/config/connect-distributed.properties
⏳ Waiting for broker coordinator on SSL://eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com:9092 ...
⏳ Waiting for KRaft broker... (attempt 1, 0s/600s)
   Debug: Last error was: [2026-02-05 09:58:40 UTC] USER=www-data EUID=0 PID=637927 ACTION=passthru ARGS=bash -c timeout 5 sudo -u kafka /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server 'eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com:9092' --command-config '/etc/kafka/client-zone-universe-main-dev-coordinator.properties' describe --status
⏳ Still waiting... (attempt 10, 59s/600s)
⏳ Still waiting... (attempt 20, 125s/600s)
⏳ Waiting for KRaft broker... (attempt 30, 190s/600s)
   Debug: Last error was: [2026-02-05 10:01:50 UTC] USER=www-data EUID=0 PID=667143 ACTION=passthru ARGS=bash -c timeout 5 sudo -u kafka /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server 'eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com:9092' --command-config '/etc/kafka/client-zone-universe-main-dev-coordinator.properties' describe --status
⏳ Still waiting... (attempt 40, 254s/600s)
⏳ Still waiting... (attempt 50, 317s/600s)
⏳ Waiting for KRaft broker... (attempt 60, 381s/600s)
   Debug: Last error was: [2026-02-05 10:05:01 UTC] USER=www-data EUID=0 PID=686396 ACTION=passthru ARGS=bash -c timeout 5 sudo -u kafka /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server 'eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com:9092' --command-config '/etc/kafka/client-zone-universe-main-dev-coordinator.properties' describe --status
⏳ Still waiting... (attempt 70, 445s/600s)
⏳ Still waiting... (attempt 80, 509s/600s)
⏳ Waiting for KRaft broker... (attempt 90, 573s/600s)
   Debug: Last error was: [2026-02-05 10:08:13 UTC] USER=www-data EUID=0 PID=705605 ACTION=passthru ARGS=bash -c timeout 5 sudo -u kafka /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server 'eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com:9092' --command-config '/etc/kafka/client-zone-universe-main-dev-coordinator.properties' describe --status
❌ Broker not reachable after 600s (605s elapsed)
[2026-02-05 10:08:45 UTC] USER=www-data EUID=0 PID=708838 ACTION=passthru ARGS=journalctl -u confluent-kraft-zone-universe-main-dev_coordinator.service -n 100 --no-pager
-- No entries --

❌ KRaft setup failed with exit code: 1

Troubleshooting:
  - Check logs: journalctl -u confluent-kraft-zone-universe-main-dev_* --no-pager -n 100
  - Verify cluster.id: cat /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/kafka_kraft_cluster_id
  - Check metadata: kafka-metadata-quorum.sh --bootstrap-server ... describe
[ERROR] ❌ Step 3 failed: 02-metadata.sh

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] Kafka setup FAILED at step: 02-metadata.sh
[ERROR] Executed 3 of 13 steps
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[ERROR] ❌ Event bus infrastructure (kafka) setup failed with exit code: 1