📊 Provisioning Job Status

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🚀 OBSERVABILITY CELL PROVISIONING STARTED
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Script: 02-observability-cell/run.sh
[INFO] Timestamp: 2026-02-05 09:38:38 UTC
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Ensuring correct permissions for observability deployment...
[2026-02-05 09:38:38 UTC] USER=www-data EUID=0 PID=570787 ACTION=fsop ARGS=chmod 775 /var/log/fastorder
[2026-02-05 09:38:38 UTC] USER=www-data EUID=0 PID=570798 ACTION=fsop ARGS=chown www-data:www-data /var/log/fastorder
[2026-02-05 09:38:38 UTC] USER=www-data EUID=0 PID=570808 ACTION=fsop ARGS=touch /var/log/fastorder/provisioning-elevated.log
[2026-02-05 09:38:38 UTC] USER=www-data EUID=0 PID=570822 ACTION=fsop ARGS=chmod 666 /var/log/fastorder/provisioning-elevated.log
[2026-02-05 09:38:38 UTC] USER=www-data EUID=0 PID=570831 ACTION=fsop ARGS=chown www-data:www-data /var/log/fastorder/provisioning-elevated.log
[OK]   Log directory: /var/log/fastorder (775)
[OK]   Log file: provisioning-elevated.log (666)
[2026-02-05 09:38:38 UTC] USER=www-data EUID=0 PID=570840 ACTION=fsop ARGS=chmod 775 /opt/fastorder/bash/scripts/env_app_setup/state
[OK]   State directory: 775
[2026-02-05 09:38:38 UTC] USER=www-data EUID=0 PID=570849 ACTION=fsop ARGS=mkdir -p /etc/fastorder/observability/certs
[2026-02-05 09:38:38 UTC] USER=www-data EUID=0 PID=570858 ACTION=fsop ARGS=chmod 750 /etc/fastorder/observability/certs
[OK]   Cert directory: /etc/fastorder/observability/certs (750 - secure)
[OK]   Lib scripts: executable (755)
[OK]   All deployment scripts: executable (755)
[OK]   All directories: accessible (755)
[OK]   ✅ All permissions verified and fixed
[CREDS] Using AWS credentials from: /var/www/.aws/credentials
[CREDS] Credential management library loaded (region: me-central-1)
[INFO] Using web-provided environment: zone-universe-main-dev
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
═══════════════════════════════════════════════════════════════════════════════
OBSERVABILITY CELL PROVISIONING
═══════════════════════════════════════════════════════════════════════════════
[INFO] Application Cell: zone-universe-main-dev
[INFO] Observability Cell: obs-zone-universe-main-dev
[INFO] Service: zone | Zone: universe | Env: dev

[INFO] Step 1/10: Provisioning network infrastructure...
[INFO]   Using existing IP for obs: 10.100.1.73
[INFO]   Using existing IP for metrics: 10.100.1.53
[INFO]   Using existing IP for dashboards: 10.100.1.194
[INFO]   Using existing IP for logstore: 10.100.1.199
[INFO]   Using existing IP for traces: 10.100.1.200
[INFO]   Using existing IP for alerts: 10.100.1.202
[INFO]   Using existing IP for telemetry: 10.100.1.203
[INFO]   Allocated observability IPs:
[INFO]     metrics: 10.100.1.53
[INFO]     alerts: 10.100.1.202
[INFO]     dashboards: 10.100.1.194
[INFO]     traces: 10.100.1.200
[INFO]     telemetry: 10.100.1.203
[INFO]     logstore: 10.100.1.199
[INFO]     proxy: 10.100.1.73
[INFO]     obs: 10.100.1.73
[ OK ] Network infrastructure allocated
[INFO] Cleaning up ports from previous environments...
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-zone-universe-main-dev
[INFO] IP Address: 10.100.1.73
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server@obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Found 19 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.73...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup completed successfully
[INFO] Configuring IP aliases on network interface...
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING NETWORK IP ALIASES
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Interface: lo
[INFO] IP Count: 8

[INFO] Configuring: metrics → 10.100.1.53
[INFO]   Configuring IP alias: 10.100.1.53/32 on lo
[OK]     ✅ IP 10.100.1.53 configured successfully on lo
[OK]     ✅ IP 10.100.1.53 verified on network interface
[INFO] Configuring: alerts → 10.100.1.202
[INFO]   Configuring IP alias: 10.100.1.202/32 on lo
[OK]     ✅ IP 10.100.1.202 configured successfully on lo
[OK]     ✅ IP 10.100.1.202 verified on network interface
[INFO] Configuring: dashboards → 10.100.1.194
[INFO]   IP 10.100.1.194 already configured on network interface
[INFO] Configuring: traces → 10.100.1.200
[INFO]   Configuring IP alias: 10.100.1.200/32 on lo
[OK]     ✅ IP 10.100.1.200 configured successfully on lo
[OK]     ✅ IP 10.100.1.200 verified on network interface
[INFO] Configuring: telemetry → 10.100.1.203
[INFO]   Configuring IP alias: 10.100.1.203/32 on lo
[OK]     ✅ IP 10.100.1.203 configured successfully on lo
[OK]     ✅ IP 10.100.1.203 verified on network interface
[INFO] Configuring: logstore → 10.100.1.199
[INFO]   Configuring IP alias: 10.100.1.199/32 on lo
[OK]     ✅ IP 10.100.1.199 configured successfully on lo
[OK]     ✅ IP 10.100.1.199 verified on network interface
[INFO] Configuring: proxy → 10.100.1.73
[INFO]   Configuring IP alias: 10.100.1.73/32 on lo
[OK]     ✅ IP 10.100.1.73 configured successfully on lo
[OK]     ✅ IP 10.100.1.73 verified on network interface
[INFO] Configuring: obs → 10.100.1.73
[INFO]   IP 10.100.1.73 already configured on network interface

[OK]   ═══════════════════════════════════════════════════════════════
[OK]   ✅ All IP aliases configured successfully
[OK]   ═══════════════════════════════════════════════════════════════
[INFO] Current IP configuration on lo:
      inet 127.0.0.1/8 scope host lo
      inet 10.100.1.186/32 scope global lo:186
      inet 10.100.1.192/32 scope global lo:pgbouncer
      inet 10.100.1.184/32 scope global lo:pgbouncer
      inet 10.100.1.204/32 scope global lo:pgbouncer
      inet 10.100.60.2/32 scope global lo
      inet 10.100.1.236/32 scope global lo
      inet 10.100.1.240/32 scope global lo
      inet 10.100.1.237/32 scope global lo
      inet 10.100.1.239/32 scope global lo
      inet 10.100.1.241/32 scope global lo
      inet 10.100.1.235/32 scope global lo
      inet 10.100.1.242/32 scope global lo
      inet 10.100.1.244/32 scope global lo:pgbouncer
      inet 10.100.1.53/32 scope global lo
      inet 10.100.1.202/32 scope global lo
      inet 10.100.1.200/32 scope global lo
      inet 10.100.1.203/32 scope global lo
      inet 10.100.1.199/32 scope global lo
      inet 10.100.1.73/32 scope global lo

[OK]   IP aliases configured on network interface
[INFO] Step 2/10: Creating DNS entries...
[INFO] Configuring DNS entries in /etc/hosts...
[INFO]   Added: metrics-zone-universe-main-dev-prometheus.fastorder.com → 10.100.1.53
[INFO]   Added: alerts-zone-universe-main-dev-alertmanager.fastorder.com → 10.100.1.202
[INFO]   Added: dashboards-zone-universe-main-dev-grafana.fastorder.com → 10.100.1.194
[INFO]   Added: traces-zone-universe-main-dev-tempo.fastorder.com → 10.100.1.200
[INFO]   Added: telemetry-zone-universe-main-dev-opentelemetry.fastorder.com → 10.100.1.203
[INFO]   Added: logstore-zone-universe-main-dev-clickhouse.fastorder.com → 10.100.1.199
[INFO]   Added: observe-zone-universe-main-dev.fastorder.com → 10.100.1.73
[INFO] Adding observability integration aliases...
[INFO]   Added alias: metrics-zone-universe-main-dev.fastorder.com → 10.100.1.53
[INFO]   Added alias: alerts-zone-universe-main-dev.fastorder.com → 10.100.1.202
[INFO]   Added alias: dashboards-zone-universe-main-dev.fastorder.com → 10.100.1.194
[INFO]   Added alias: traces-zone-universe-main-dev.fastorder.com → 10.100.1.200
[INFO]   Added alias: telemetry-zone-universe-main-dev.fastorder.com → 10.100.1.203
[INFO]   Added alias: logstore-zone-universe-main-dev.fastorder.com → 10.100.1.199
[2026-02-05 09:38:41 UTC] USER=www-data EUID=0 PID=571542 ACTION=fsop ARGS=sed -i /observe-zone-universe-main-dev.fastorder.com/d /etc/hosts
[INFO]   Added alias: observe-zone-universe-main-dev.fastorder.com → 10.100.1.73
[OK]   DNS entries created
[INFO] Step 3/10: Creating AWS Secrets Manager structure...
[INFO] Creating AWS Secrets Manager structure
[INFO]   Base path: fastorder/observability/zone/universe/dev
[INFO]   Observability Cell: obs-zone-universe-main-dev
[INFO]   Application Cell: zone-universe-main-dev
[INFO]   Creating: fastorder/observability/zone/universe/dev/metrics
[INFO]   Creating: fastorder/observability/zone/universe/dev/dashboards
[INFO]   Creating: fastorder/observability/zone/universe/dev/logstore
[INFO]   Creating: fastorder/observability/zone/universe/dev/traces
[INFO]   Creating: fastorder/observability/zone/universe/dev/telemetry
[INFO]   Creating: fastorder/observability/zone/universe/dev/alerts
[INFO] Secrets structure created successfully
[OK]   Secrets structure created
[INFO] Step 4/10: Generating mTLS certificates...
[INFO] Generating mTLS certificates for observability cell
[INFO]   Observability Cell: obs-zone-universe-main-dev
[INFO]   Components: prometheus,grafana,loki,tempo,otlp_collector,clickhouse,alertmanager
[INFO]   Creating certificate directory: /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[2026-02-05 09:39:04 UTC] USER=www-data EUID=0 PID=572428 ACTION=fsop ARGS=mkdir -p /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[2026-02-05 09:39:04 UTC] USER=www-data EUID=0 PID=572437 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[INFO]   Generating CA certificate for obs-zone-universe-main-dev
[2026-02-05 09:39:04 UTC] USER=www-data EUID=0 PID=572447 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-key.pem 4096
[2026-02-05 09:39:07 UTC] USER=www-data EUID=0 PID=572543 ACTION=fsop ARGS=openssl req -new -x509 -days 3650 -key /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-key.pem -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=obs-zone-universe-main-dev-ca
[2026-02-05 09:39:07 UTC] USER=www-data EUID=0 PID=572552 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-key.pem
[2026-02-05 09:39:07 UTC] USER=www-data EUID=0 PID=572561 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem
[INFO]   CA certificate created: /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem
[INFO]   Generating certificate for: prometheus
[2026-02-05 09:39:07 UTC] USER=www-data EUID=0 PID=572570 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/prometheus-key.pem 2048
[2026-02-05 09:39:07 UTC] USER=www-data EUID=0 PID=572609 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-zone-universe-main-dev/prometheus-key.pem -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/prometheus-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=prometheus.obs-zone-universe-main-dev
[2026-02-05 09:39:07 UTC] USER=www-data EUID=0 PID=572618 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-zone-universe-main-dev/prometheus-csr.pem -CA /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/prometheus-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = prometheus.obs-zone-universe-main-dev
[2026-02-05 09:39:07 UTC] USER=www-data EUID=0 PID=572627 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/prometheus-key.pem
[2026-02-05 09:39:07 UTC] USER=www-data EUID=0 PID=572636 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/prometheus-cert.pem
[2026-02-05 09:39:07 UTC] USER=www-data EUID=0 PID=572645 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-zone-universe-main-dev/prometheus-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-zone-universe-main-dev/prometheus-cert.pem
[INFO]   Generating certificate for: grafana
[2026-02-05 09:39:07 UTC] USER=www-data EUID=0 PID=572654 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/grafana-key.pem 2048
[2026-02-05 09:39:07 UTC] USER=www-data EUID=0 PID=572665 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-zone-universe-main-dev/grafana-key.pem -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/grafana-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=grafana.obs-zone-universe-main-dev
[2026-02-05 09:39:07 UTC] USER=www-data EUID=0 PID=572674 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-zone-universe-main-dev/grafana-csr.pem -CA /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/grafana-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = grafana.obs-zone-universe-main-dev
[2026-02-05 09:39:08 UTC] USER=www-data EUID=0 PID=572683 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/grafana-key.pem
[2026-02-05 09:39:08 UTC] USER=www-data EUID=0 PID=572692 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/grafana-cert.pem
[2026-02-05 09:39:08 UTC] USER=www-data EUID=0 PID=572701 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-zone-universe-main-dev/grafana-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-zone-universe-main-dev/grafana-cert.pem
[INFO]   Generating certificate for: loki
[2026-02-05 09:39:08 UTC] USER=www-data EUID=0 PID=572710 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/loki-key.pem 2048
[2026-02-05 09:39:08 UTC] USER=www-data EUID=0 PID=572719 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-zone-universe-main-dev/loki-key.pem -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/loki-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=loki.obs-zone-universe-main-dev
[2026-02-05 09:39:08 UTC] USER=www-data EUID=0 PID=572728 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-zone-universe-main-dev/loki-csr.pem -CA /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/loki-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = loki.obs-zone-universe-main-dev
[2026-02-05 09:39:08 UTC] USER=www-data EUID=0 PID=572737 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/loki-key.pem
[2026-02-05 09:39:08 UTC] USER=www-data EUID=0 PID=572746 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/loki-cert.pem
[2026-02-05 09:39:08 UTC] USER=www-data EUID=0 PID=572755 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-zone-universe-main-dev/loki-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-zone-universe-main-dev/loki-cert.pem
[INFO]   Generating certificate for: tempo
[2026-02-05 09:39:08 UTC] USER=www-data EUID=0 PID=572764 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/tempo-key.pem 2048
[2026-02-05 09:39:08 UTC] USER=www-data EUID=0 PID=572773 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-zone-universe-main-dev/tempo-key.pem -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/tempo-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=tempo.obs-zone-universe-main-dev
[2026-02-05 09:39:08 UTC] USER=www-data EUID=0 PID=572782 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-zone-universe-main-dev/tempo-csr.pem -CA /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/tempo-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = tempo.obs-zone-universe-main-dev
[2026-02-05 09:39:08 UTC] USER=www-data EUID=0 PID=572791 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/tempo-key.pem
[2026-02-05 09:39:08 UTC] USER=www-data EUID=0 PID=572800 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/tempo-cert.pem
[2026-02-05 09:39:08 UTC] USER=www-data EUID=0 PID=572809 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-zone-universe-main-dev/tempo-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-zone-universe-main-dev/tempo-cert.pem
[INFO]   Generating certificate for: otlp_collector
[2026-02-05 09:39:08 UTC] USER=www-data EUID=0 PID=572818 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/otlp_collector-key.pem 2048
[2026-02-05 09:39:09 UTC] USER=www-data EUID=0 PID=572836 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-zone-universe-main-dev/otlp_collector-key.pem -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/otlp_collector-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=otlp_collector.obs-zone-universe-main-dev
[2026-02-05 09:39:09 UTC] USER=www-data EUID=0 PID=572845 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-zone-universe-main-dev/otlp_collector-csr.pem -CA /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/otlp_collector-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = otlp_collector.obs-zone-universe-main-dev
[2026-02-05 09:39:09 UTC] USER=www-data EUID=0 PID=572854 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/otlp_collector-key.pem
[2026-02-05 09:39:09 UTC] USER=www-data EUID=0 PID=572866 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/otlp_collector-cert.pem
[2026-02-05 09:39:09 UTC] USER=www-data EUID=0 PID=572875 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-zone-universe-main-dev/otlp_collector-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-zone-universe-main-dev/otlp_collector-cert.pem
[INFO]   Generating certificate for: clickhouse
[2026-02-05 09:39:09 UTC] USER=www-data EUID=0 PID=572888 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/clickhouse-key.pem 2048
[2026-02-05 09:39:09 UTC] USER=www-data EUID=0 PID=572911 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-zone-universe-main-dev/clickhouse-key.pem -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/clickhouse-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=clickhouse.obs-zone-universe-main-dev
[2026-02-05 09:39:09 UTC] USER=www-data EUID=0 PID=572920 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-zone-universe-main-dev/clickhouse-csr.pem -CA /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/clickhouse-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = clickhouse.obs-zone-universe-main-dev
[2026-02-05 09:39:09 UTC] USER=www-data EUID=0 PID=572929 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/clickhouse-key.pem
[2026-02-05 09:39:09 UTC] USER=www-data EUID=0 PID=572938 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/clickhouse-cert.pem
[2026-02-05 09:39:09 UTC] USER=www-data EUID=0 PID=572948 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-zone-universe-main-dev/clickhouse-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-zone-universe-main-dev/clickhouse-cert.pem
[INFO]   Generating certificate for: alertmanager
[2026-02-05 09:39:09 UTC] USER=www-data EUID=0 PID=572957 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/alertmanager-key.pem 2048
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=572971 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-zone-universe-main-dev/alertmanager-key.pem -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/alertmanager-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=alertmanager.obs-zone-universe-main-dev
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573002 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-zone-universe-main-dev/alertmanager-csr.pem -CA /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/alertmanager-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = alertmanager.obs-zone-universe-main-dev
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573024 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/alertmanager-key.pem
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573034 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/alertmanager-cert.pem
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573047 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-zone-universe-main-dev/alertmanager-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-zone-universe-main-dev/alertmanager-cert.pem
[INFO]   Generating PHP client certificate for metrics service...
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573057 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/php-client-key.pem 2048
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573068 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-zone-universe-main-dev/php-client-key.pem -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/php-client-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Dashboard/CN=php-metrics-client.obs-zone-universe-main-dev
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573077 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-zone-universe-main-dev/php-client-csr.pem -CA /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/php-client-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Dashboard, CN = php-metrics-client.obs-zone-universe-main-dev
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573086 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/php-client-key.pem
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573095 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/php-client-cert.pem
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573104 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-zone-universe-main-dev/php-client-key.pem
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573113 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-zone-universe-main-dev/php-client-cert.pem
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573122 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-zone-universe-main-dev/php-client-csr.pem
[INFO]   PHP client certificate created: /etc/fastorder/observability/certs/obs-zone-universe-main-dev/php-client-cert.pem
[INFO]   Generating Apache client certificate for mTLS reverse proxy...
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573131 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/apache-client-key.pem 2048
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573144 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-zone-universe-main-dev/apache-client-key.pem -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/apache-client-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=ReverseProxy/CN=apache-proxy.obs-zone-universe-main-dev
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573153 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-zone-universe-main-dev/apache-client-csr.pem -CA /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-zone-universe-main-dev/apache-client-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = ReverseProxy, CN = apache-proxy.obs-zone-universe-main-dev
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573180 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/apache-client-key.pem
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573189 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/apache-client-combined.pem
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573198 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/apache-client-cert.pem
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573207 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-zone-universe-main-dev/apache-client-key.pem
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573216 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-zone-universe-main-dev/apache-client-cert.pem
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573225 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-zone-universe-main-dev/apache-client-combined.pem
[2026-02-05 09:39:10 UTC] USER=www-data EUID=0 PID=573234 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-zone-universe-main-dev/apache-client-csr.pem
[INFO]   Apache client certificate created: /etc/fastorder/observability/certs/obs-zone-universe-main-dev/apache-client-cert.pem
[INFO]   Apache combined cert+key: /etc/fastorder/observability/certs/obs-zone-universe-main-dev/apache-client-combined.pem
[INFO]   Storing mTLS certificates in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/zone/universe/main/dev/mtls/php-client-z94quZ",
    "Name": "fastorder/observability/zone/universe/main/dev/mtls/php-client",
    "VersionId": "01c7a325-02b2-4bb7-9d43-8183dd0b6d8a"
}
[INFO]   mTLS certificates stored in Secrets Manager: fastorder/observability/zone/universe/main/dev/mtls/php-client
[INFO] mTLS certificates generated successfully
[INFO]   Certificate directory: /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[INFO]   PHP client cert: /etc/fastorder/observability/certs/obs-zone-universe-main-dev/php-client-cert.pem
[INFO]   PHP client key: /etc/fastorder/observability/certs/obs-zone-universe-main-dev/php-client-key.pem
[INFO]   Apache client cert: /etc/fastorder/observability/certs/obs-zone-universe-main-dev/apache-client-cert.pem
[INFO]   Apache combined (for SSLProxyMachineCertificateFile): /etc/fastorder/observability/certs/obs-zone-universe-main-dev/apache-client-combined.pem
[OK]   mTLS certificates generated
[INFO] Step 5/10: Deploying log storage backend...
[INFO]   Provider: clickhouse (selected)
[INFO]   Note: Deployed before telemetry (OtelCol depends on log storage)
[INFO]   FQDN: logstore-zone-universe-main-dev-clickhouse.fastorder.com
[INFO]   IP: 10.100.1.199
[INFO] Deploying log backend: clickhouse...
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] LOG STORAGE BACKEND DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: clickhouse
[INFO] Observability Cell: obs-zone-universe-main-dev
[INFO] FQDN: logstore-zone-universe-main-dev-clickhouse.fastorder.com
[INFO] IP: 10.100.1.199
[INFO] S3 Bucket: fastorder-logs-universe-dev
[INFO] Retention: 90 days
[INFO] ═══════════════════════════════════════════════════════════════

[2026-02-05 09:39:12 UTC] USER=unknown EUID=33 PID=573288 ACTION=fsop ARGS=chmod +x /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh
/bin/chmod: changing permissions of '/opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh': Operation not permitted
[INFO] Using provider: clickhouse
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=zone, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-zone-universe-main-dev...
[2026-02-05 09:39:12 UTC] USER=www-data EUID=0 PID=573305 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-02-05 09:39:12 UTC] USER=www-data EUID=0 PID=573314 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-02-05 09:39:12 UTC] USER=www-data EUID=0 PID=573323 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-02-05 09:39:12 UTC] USER=www-data EUID=0 PID=573332 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-zone-universe-main-dev
[INFO] IP Address: 10.100.1.199
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server@obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Found 19 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.199...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[INFO] Binding ClickHouse to allocated IP: 10.100.1.199
[INFO] Deploying ClickHouse for obs-zone-universe-main-dev
[INFO]   FQDN: logstore-zone-universe-main-dev-clickhouse.fastorder.com
[INFO]   Allocated IP: 10.100.1.199
[INFO]   VM IP: 10.100.1.199
[INFO]   Ports: HTTP=8123 TCP=9000 Interserver=9009
[INFO]   S3 Bucket: fastorder-logs-universe-dev (region=me-central-1)
[INFO]   Retention: 90 days
[INFO] Checking if ClickHouse is installed...
[OK]   ClickHouse already installed
[2026-02-05 09:39:13 UTC] USER=www-data EUID=0 PID=573565 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-server-obs-zone-universe-main-dev/config.d
[2026-02-05 09:39:13 UTC] USER=www-data EUID=0 PID=573574 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-server-obs-zone-universe-main-dev/users.d
[2026-02-05 09:39:13 UTC] USER=www-data EUID=0 PID=573583 ACTION=fsop ARGS=mkdir -p /var/lib/clickhouse-obs-zone-universe-main-dev
[2026-02-05 09:39:13 UTC] USER=www-data EUID=0 PID=573592 ACTION=fsop ARGS=mkdir -p /var/log/clickhouse-server-obs-zone-universe-main-dev
[2026-02-05 09:39:13 UTC] USER=www-data EUID=0 PID=573602 ACTION=passthru ARGS=chmod 755 /etc/clickhouse-server-obs-zone-universe-main-dev
[2026-02-05 09:39:13 UTC] USER=www-data EUID=0 PID=573614 ACTION=passthru ARGS=chmod 700 /var/lib/clickhouse-obs-zone-universe-main-dev
[2026-02-05 09:39:13 UTC] USER=www-data EUID=0 PID=573623 ACTION=passthru ARGS=chmod 750 /var/log/clickhouse-server-obs-zone-universe-main-dev
[INFO] No existing logs_writer credentials found - generating new ones
[INFO] No existing metrics_reader credentials found - generating new ones
[INFO] TLS configuration exported for clickhouse
[INFO]   Cert: /etc/fastorder/observability/certs/obs-zone-universe-main-dev/clickhouse-cert.pem
[INFO]   Key:  /etc/fastorder/observability/certs/obs-zone-universe-main-dev/clickhouse-key.pem
[INFO]   CA:   /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem
[INFO] Configuring certificate permissions for clickhouse (user: clickhouse)
[INFO] Initializing certificate directory for obs-zone-universe-main-dev...
[2026-02-05 09:39:17 UTC] USER=www-data EUID=0 PID=573842 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-02-05 09:39:17 UTC] USER=www-data EUID=0 PID=573859 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-02-05 09:39:17 UTC] USER=www-data EUID=0 PID=573879 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[INFO]   Setting file permissions...
[2026-02-05 09:39:17 UTC] USER=www-data EUID=0 PID=573889 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/clickhouse-cert.pem
[2026-02-05 09:39:17 UTC] USER=www-data EUID=0 PID=573898 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem
[2026-02-05 09:39:17 UTC] USER=www-data EUID=0 PID=573907 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/clickhouse-key.pem
[INFO]   Setting file ownership...
[2026-02-05 09:39:17 UTC] USER=www-data EUID=0 PID=573916 ACTION=passthru ARGS=chown root:clickhouse /etc/fastorder/observability/certs/obs-zone-universe-main-dev/clickhouse-key.pem
[2026-02-05 09:39:17 UTC] USER=www-data EUID=0 PID=573925 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-zone-universe-main-dev/clickhouse-cert.pem /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for clickhouse
[INFO] Creating ClickHouse configuration...
[2026-02-05 09:39:17 UTC] USER=www-data EUID=0 PID=573999 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /etc/clickhouse-server-obs-zone-universe-main-dev
[2026-02-05 09:39:17 UTC] USER=www-data EUID=0 PID=574008 ACTION=passthru ARGS=bash -c chmod 640 /etc/clickhouse-server-obs-zone-universe-main-dev/*.xml
[OK]   ClickHouse configuration created
[INFO] Creating logs table schema...
[2026-02-05 09:39:17 UTC] USER=www-data EUID=0 PID=574026 ACTION=passthru ARGS=sed -i s/__RETENTION_DAYS__/90/g /etc/clickhouse-server-obs-zone-universe-main-dev/logs_schema.sql
[OK]   Logs schema created
[INFO] Creating systemd service...
[2026-02-05 09:39:18 UTC] USER=www-data EUID=0 PID=574055 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /var/lib/clickhouse-obs-zone-universe-main-dev
[2026-02-05 09:39:18 UTC] USER=www-data EUID=0 PID=574064 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /var/log/clickhouse-server-obs-zone-universe-main-dev
[OK]   Systemd service created
[INFO] Starting ClickHouse service...
[2026-02-05 09:39:18 UTC] USER=www-data EUID=0 PID=574083 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 09:39:18 UTC] USER=www-data EUID=0 PID=574143 ACTION=passthru ARGS=systemctl enable clickhouse-server-obs-zone-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/clickhouse-server-obs-zone-universe-main-dev.service → /etc/systemd/system/clickhouse-server-obs-zone-universe-main-dev.service.
[2026-02-05 09:39:19 UTC] USER=www-data EUID=0 PID=574198 ACTION=passthru ARGS=systemctl start clickhouse-server-obs-zone-universe-main-dev.service
[INFO] Waiting for ClickHouse to be ready...
[OK]   ClickHouse is ready
[INFO] Initializing database schema...
[OK]   Schema initialized
[INFO] Storing ClickHouse credentials in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/zone/universe/main/dev/clickhouse/server/logs_writer-KQnkBL",
    "Name": "fastorder/observability/zone/universe/main/dev/clickhouse/server/logs_writer",
    "VersionId": "70add8c2-ff1a-421d-8f8e-53e2ff39c5de"
}
[OK]   logs_writer credentials stored and verified in Secrets Manager
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/zone/universe/main/dev/clickhouse/server/metrics_reader-79BRXN",
    "Name": "fastorder/observability/zone/universe/main/dev/clickhouse/server/metrics_reader",
    "VersionId": "d41cc209-e49a-4b45-9e25-aa4f25b0b30c"
}
[OK]   metrics_reader credentials stored and verified in Secrets Manager
[INFO] Validating ClickHouse deployment...
[INFO] ClickHouse version: 25.10.1.3832
[INFO] Tables created: .inner_id.8cc84eb1-934a-4f7a-a665-cc87c22d8488
.inner_id.ae8bfa2e-2841-429b-9c17-cc91a6593874
application_logs
error_logs_mv
iam_audit_event
metrics_all
otel_logs
request_logs_mv
security_access
[INFO] Test log inserted. Total logs: 1
[OK]   ✅ ClickHouse deployment validated
[INFO] Setting up clickhouse-backup for backup management...
[OK]   clickhouse-backup already installed
[INFO] Creating clickhouse-backup configuration...
[2026-02-05 09:39:32 UTC] USER=www-data EUID=0 PID=575421 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-backup
[2026-02-05 09:39:32 UTC] USER=www-data EUID=0 PID=575431 ACTION=passthru ARGS=chown root:clickhouse /etc/clickhouse-backup
[2026-02-05 09:39:32 UTC] USER=www-data EUID=0 PID=575447 ACTION=passthru ARGS=chmod 750 /etc/clickhouse-backup
[2026-02-05 09:39:32 UTC] USER=www-data EUID=0 PID=575472 ACTION=passthru ARGS=chown root:clickhouse /etc/clickhouse-backup/config-obs-zone-universe-main-dev.yml
[2026-02-05 09:39:32 UTC] USER=www-data EUID=0 PID=575492 ACTION=passthru ARGS=chmod 640 /etc/clickhouse-backup/config-obs-zone-universe-main-dev.yml
[2026-02-05 09:39:32 UTC] USER=www-data EUID=0 PID=575549 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 09:39:33 UTC] USER=www-data EUID=0 PID=575605 ACTION=passthru ARGS=systemctl enable clickhouse-backup-api-obs-zone-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/clickhouse-backup-api-obs-zone-universe-main-dev.service → /etc/systemd/system/clickhouse-backup-api-obs-zone-universe-main-dev.service.
[2026-02-05 09:39:33 UTC] USER=www-data EUID=0 PID=575652 ACTION=passthru ARGS=systemctl start clickhouse-backup-api-obs-zone-universe-main-dev.service
[2026-02-05 09:39:34 UTC] USER=www-data EUID=0 PID=575662 ACTION=passthru ARGS=systemctl enable clickhouse-backup@obs-zone-universe-main-dev.timer
Created symlink /etc/systemd/system/timers.target.wants/clickhouse-backup@obs-zone-universe-main-dev.timer → /etc/systemd/system/clickhouse-backup@.timer.
[2026-02-05 09:39:34 UTC] USER=www-data EUID=0 PID=575717 ACTION=passthru ARGS=systemctl start clickhouse-backup@obs-zone-universe-main-dev.timer
[OK]   clickhouse-backup configured and started
[INFO] Setting up ClickHouse exporter for Prometheus...
[OK]   clickhouse_exporter already installed
[2026-02-05 09:39:34 UTC] USER=www-data EUID=0 PID=575739 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 09:39:35 UTC] USER=www-data EUID=0 PID=575791 ACTION=passthru ARGS=systemctl enable clickhouse_exporter-obs-zone-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/clickhouse_exporter-obs-zone-universe-main-dev.service → /etc/systemd/system/clickhouse_exporter-obs-zone-universe-main-dev.service.
[2026-02-05 09:39:36 UTC] USER=www-data EUID=0 PID=575868 ACTION=passthru ARGS=systemctl start clickhouse_exporter-obs-zone-universe-main-dev.service
[OK]   clickhouse_exporter configured and started

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ ClickHouse Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   FQDN: logstore-zone-universe-main-dev-clickhouse.fastorder.com
[INFO]   IP: 10.100.1.199
[INFO]   HTTP Port: 8123
[INFO]   Native Port: 9000
[INFO]   Database: logs
[INFO]   Retention: 90 days
[INFO]   Storage: Tiered (Local → S3: fastorder-logs-universe-dev in me-central-1)
[INFO] 
[INFO] Backup & Monitoring:
[INFO]   clickhouse-backup API: http://10.100.1.199:7171
[INFO]   clickhouse_exporter: http://10.100.1.199:9116/metrics
[INFO]   Backup Schedule: Daily at 2:00 AM
[INFO]   Local Backups Retained: 7
[INFO] 
[INFO] Credentials stored in AWS Secrets Manager:
[INFO]   Writers: fastorder/observability/zone/universe/main/dev/clickhouse/server/logs_writer
[INFO]   Readers: fastorder/observability/zone/universe/main/dev/clickhouse/server/metrics_reader (for PHP metrics service)
[INFO] 
[INFO] Example queries (using credentials from Secrets Manager):
[INFO]   # Write logs:
[INFO]   clickhouse-client --host logstore-zone-universe-main-dev-clickhouse.fastorder.com --port 9000 --user logs_writer --password '***' --query 'SELECT 1'
[INFO] 
[INFO]   # Read metrics (PHP metrics service):
[INFO]   clickhouse-client --host logstore-zone-universe-main-dev-clickhouse.fastorder.com --port 9000 --user metrics_reader --password '***' --query 'SELECT * FROM system.metrics'
[INFO] 
[INFO] HTTPS Setup (run on web-03/skeleton server):
[INFO]   # Set up HTTPS reverse proxy with Let's Encrypt:
[INFO]   OBS_CELL=obs-zone-universe-main-dev BACKEND_IP=10.100.1.199 sudo bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/../https/setup-clickhouse-https.sh
[INFO] 
[INFO]   # Or add --setup-https flag when running this script
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Log Storage Backend Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: clickhouse
[INFO] FQDN: logstore-zone-universe-main-dev-clickhouse.fastorder.com
[INFO] IP: 10.100.1.199
[INFO] Retention: 90 days
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering ClickHouse in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       ClickHouse
[INFO]   Identifier:        zone-universe-main-dev-clickhouse
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.199
[INFO]   Port:              8443
[INFO]   FQDN:              logstore-zone-universe-main-dev-clickhouse.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ INVALID REQUEST
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] Response: {"success":false,"error":"Invalid JSON: Control character error, possibly incorrectly encoded"}
[ERROR] 
[ERROR] Request payload:
  {
    "env_id": "zone-universe-main-dev",
    "application": "ClickHouse",
    "identifier": "zone-universe-main-dev-clickhouse",
    "identifier_parent": "cluster",
    "ip": "10.100.1.199",
    "port": 8443,
    "fqdn": "logstore-zone-universe-main-dev-clickhouse.fastorder.com",
    "status": "running",
    "meta": {
      "role": "log_storage",
      "provider": "clickhouse",
      "version": "25.10
  1.3832",
      "http_port": 8123,
      "native_port": 9000,
      "https_port": 8443,
      "protocol": "https",
      "metrics_enabled": true,
      "metrics_port": 8123,
      "metrics_path": "/metrics",
      "health_endpoint": "https://logstore-zone-universe-main-dev-clickhouse.fastorder.com/ping",
      "retention_days": 90,
      "s3_bucket": "fastorder-logs-universe-dev"
  }
  }
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[WARN] ⚠️  Failed to register ClickHouse (service is running)
[OK]   clickhouse deployed successfully
[OK]   Log storage backend deployed
[INFO] Step 6/10: Deploying telemetry collector...
[INFO]   Provider: otlp (backend implementation - internal)
[INFO]   Endpoint: telemetry-zone-universe-main-dev-opentelemetry.fastorder.com (stable, exposed to clients)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] TELEMETRY COLLECTOR DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: otlp
[INFO] Observability Cell: obs-zone-universe-main-dev
[INFO] FQDN: telemetry-zone-universe-main-dev-opentelemetry.fastorder.com
[INFO] IP: 10.100.1.203
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: otlp
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Telemetry/provider/otlp.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=zone, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-zone-universe-main-dev...
[2026-02-05 09:39:37 UTC] USER=www-data EUID=0 PID=575962 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-02-05 09:39:37 UTC] USER=www-data EUID=0 PID=575980 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-zone-universe-main-dev
[INFO] IP Address: 10.100.1.203
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-zone-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service clickhouse-server@obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server@obs-zone-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Found 21 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.203...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[INFO] Binding to allocated IP: 10.100.1.203
[INFO] Deploying OpenTelemetry Collector for observability cell: obs-zone-universe-main-dev
[INFO] FQDN:         telemetry-zone-universe-main-dev-opentelemetry.fastorder.com
[INFO] Allocated IP: 10.100.1.203
[INFO] VM IP:        10.100.1.203
[INFO] Ports:        gRPC=4317 HTTP=4318 Metrics=8888 Prom=8889
[VERSION] Fetching latest version for otel_collector from GitHub (open-telemetry/opentelemetry-collector-releases)...
[VERSION] Latest otel_collector version: 0.145.0
[INFO] Resolved OpenTelemetry Collector version: 0.145.0
[OK]   User 'otelcol' already exists
[INFO] Checking if OpenTelemetry Collector is installed...
[OK]   OpenTelemetry Collector already installed at /usr/local/bin/otelcol-contrib
[INFO] Creating configuration/data directories...
[2026-02-05 09:39:39 UTC] USER=www-data EUID=0 PID=576287 ACTION=passthru ARGS=mkdir -p /etc/otelcol/obs-zone-universe-main-dev
[2026-02-05 09:39:39 UTC] USER=www-data EUID=0 PID=576296 ACTION=passthru ARGS=mkdir -p /var/lib/otelcol/obs-zone-universe-main-dev
[2026-02-05 09:39:39 UTC] USER=www-data EUID=0 PID=576305 ACTION=passthru ARGS=chown -R otelcol:otelcol /etc/otelcol/obs-zone-universe-main-dev /var/lib/otelcol/obs-zone-universe-main-dev
[2026-02-05 09:39:39 UTC] USER=www-data EUID=0 PID=576314 ACTION=passthru ARGS=chmod 0750 /etc/otelcol/obs-zone-universe-main-dev
[2026-02-05 09:39:39 UTC] USER=www-data EUID=0 PID=576323 ACTION=passthru ARGS=chmod 0750 /var/lib/otelcol/obs-zone-universe-main-dev
[INFO] Retrieving ClickHouse credentials from Secrets Manager...
[OK]   Retrieved ClickHouse credentials from Secrets Manager
[INFO] Creating OpenTelemetry Collector configuration...
[INFO] ClickHouse exporter enabled: tcp://logstore-zone-universe-main-dev-clickhouse.fastorder.com:9000
[2026-02-05 09:39:41 UTC] USER=www-data EUID=0 PID=576422 ACTION=passthru ARGS=chown otelcol:otelcol /etc/otelcol/obs-zone-universe-main-dev/config.yaml
[2026-02-05 09:39:41 UTC] USER=www-data EUID=0 PID=576431 ACTION=passthru ARGS=chmod 0640 /etc/otelcol/obs-zone-universe-main-dev/config.yaml
[OK]   Configuration created at /etc/otelcol/obs-zone-universe-main-dev/config.yaml
[INFO] Setting up TLS certificate permissions...
[INFO] Configuring certificate permissions for otlp_collector (user: otelcol)
[INFO] Initializing certificate directory for obs-zone-universe-main-dev...
[2026-02-05 09:39:41 UTC] USER=www-data EUID=0 PID=576440 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-02-05 09:39:41 UTC] USER=www-data EUID=0 PID=576449 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-02-05 09:39:41 UTC] USER=www-data EUID=0 PID=576458 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-02-05 09:39:41 UTC] USER=www-data EUID=0 PID=576467 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[INFO]   Setting file permissions...
[2026-02-05 09:39:41 UTC] USER=www-data EUID=0 PID=576477 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/otlp_collector-cert.pem
[2026-02-05 09:39:41 UTC] USER=www-data EUID=0 PID=576486 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem
[2026-02-05 09:39:41 UTC] USER=www-data EUID=0 PID=576495 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/otlp_collector-key.pem
[INFO]   Setting file ownership...
[2026-02-05 09:39:41 UTC] USER=www-data EUID=0 PID=576504 ACTION=passthru ARGS=chown root:otelcol /etc/fastorder/observability/certs/obs-zone-universe-main-dev/otlp_collector-key.pem
[2026-02-05 09:39:41 UTC] USER=www-data EUID=0 PID=576513 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-zone-universe-main-dev/otlp_collector-cert.pem /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for otlp_collector
[OK]   Certificate permissions configured
[INFO] Creating systemd service: otelcol-obs-zone-universe-main-dev
[OK]   Systemd service created at /etc/systemd/system/otelcol-obs-zone-universe-main-dev.service
[INFO] Adding /etc/hosts entry for telemetry-zone-universe-main-dev-opentelemetry.fastorder.com -> 10.100.1.203
[2026-02-05 09:39:41 UTC] USER=www-data EUID=0 PID=576533 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*telemetry-zone-universe-main-dev-opentelemetry.fastorder.com/10.100.1.203    telemetry-zone-universe-main-dev-opentelemetry.fastorder.com/ /etc/hosts
[OK]   Updated /etc/hosts entry to use VM_IP
[INFO] Storing OTLP configuration metadata in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/zone/universe/main/dev/otlp/collector-GRW6jm",
    "Name": "fastorder/observability/zone/universe/main/dev/otlp/collector",
    "VersionId": "5c9d0927-b8c8-4b98-b1e6-5902a45d59c6"
}
[OK]   Configuration metadata stored/updated in AWS Secrets Manager: fastorder/observability/zone/universe/main/dev/otlp/collector
[INFO] Enabling and starting OpenTelemetry Collector service...
[2026-02-05 09:39:43 UTC] USER=www-data EUID=0 PID=576588 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 09:39:44 UTC] USER=www-data EUID=0 PID=576637 ACTION=passthru ARGS=systemctl enable otelcol-obs-zone-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/otelcol-obs-zone-universe-main-dev.service → /etc/systemd/system/otelcol-obs-zone-universe-main-dev.service.
[2026-02-05 09:39:44 UTC] USER=www-data EUID=0 PID=576690 ACTION=passthru ARGS=systemctl restart otelcol-obs-zone-universe-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-02-05 09:39:47 UTC] USER=www-data EUID=0 PID=576787 ACTION=passthru ARGS=systemctl is-active --quiet otelcol-obs-zone-universe-main-dev.service
[OK]   ✅ OpenTelemetry Collector is running
[OK]   ✅ gRPC endpoint listening on port 4317
[OK]   ✅ HTTP endpoint listening on port 4318
[OK]   ✅ Prometheus metrics endpoint listening on port 8889
[INFO] Service logs (last 10 lines):
[2026-02-05 09:39:48 UTC] USER=www-data EUID=0 PID=576835 ACTION=passthru ARGS=journalctl -u otelcol-obs-zone-universe-main-dev.service -n 10 --no-pager
Feb 05 09:39:45 web-03 otelcol-obs-zone-universe-main-dev[576701]: 2026-02-05T09:39:45.255Z        info        internal/resourcedetection.go:125        began detecting resource information        {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics"}
Feb 05 09:39:45 web-03 otelcol-obs-zone-universe-main-dev[576701]: 2026-02-05T09:39:45.257Z        info        system/system.go:201        This attribute changed from int to string. Temporarily switch back to int using the feature gate.        {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics", "attribute": "host.cpu.family", "feature gate": "processor.resourcedetection.hostCPUModelAndFamilyAsString"}
Feb 05 09:39:45 web-03 otelcol-obs-zone-universe-main-dev[576701]: 2026-02-05T09:39:45.258Z        info        system/system.go:220        This attribute changed from int to string. Temporarily switch back to int using the feature gate.        {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics", "attribute": "host.cpu.model.id", "feature gate": "processor.resourcedetection.hostCPUModelAndFamilyAsString"}
Feb 05 09:39:45 web-03 otelcol-obs-zone-universe-main-dev[576701]: 2026-02-05T09:39:45.258Z        info        internal/resourcedetection.go:139        detected resource information        {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics", "resource": {"host.name":"web-03","os.type":"linux"}}
Feb 05 09:39:45 web-03 otelcol-obs-zone-universe-main-dev[576701]: 2026-02-05T09:39:45.259Z        info        otlpreceiver@v0.91.0/otlp.go:83        Starting GRPC server        {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "10.100.1.203:4317"}
Feb 05 09:39:45 web-03 otelcol-obs-zone-universe-main-dev[576701]: 2026-02-05T09:39:45.260Z        info        otlpreceiver@v0.91.0/otlp.go:101        Starting HTTP server        {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "10.100.1.203:4318"}
Feb 05 09:39:45 web-03 otelcol-obs-zone-universe-main-dev[576701]: 2026-02-05T09:39:45.306Z        info        prometheusreceiver@v0.91.0/metrics_receiver.go:231        Scrape job added        {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "jobName": "otel-collector"}
Feb 05 09:39:45 web-03 otelcol-obs-zone-universe-main-dev[576701]: 2026-02-05T09:39:45.306Z        info        prometheusreceiver@v0.91.0/metrics_receiver.go:240        Starting discovery manager        {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
Feb 05 09:39:45 web-03 otelcol-obs-zone-universe-main-dev[576701]: 2026-02-05T09:39:45.307Z        info        service@v0.91.0/service.go:171        Everything is ready. Begin running and processing data.
Feb 05 09:39:45 web-03 otelcol-obs-zone-universe-main-dev[576701]: 2026-02-05T09:39:45.307Z        info        prometheusreceiver@v0.91.0/metrics_receiver.go:282        Starting scrape manager        {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Telemetry Collector Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: otlp
[INFO] FQDN: telemetry-zone-universe-main-dev-opentelemetry.fastorder.com
[INFO] IP: 10.100.1.203
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering OpenTelemetry Collector in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       OpenTelemetry Collector
[INFO]   Identifier:        zone-universe-main-dev-opentelemetry
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.203
[INFO]   Port:              4317
[INFO]   FQDN:              telemetry-zone-universe-main-dev-opentelemetry.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: e5f663b5-18dc-450d-bcb2-a582c5fe642b
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   ✅ OpenTelemetry Collector registered in dashboard
[INFO] Setting up OpenTelemetry Collector metrics collection timer...
[2026-02-05 09:39:48 UTC] USER=www-data EUID=0 PID=576918 ACTION=passthru ARGS=mv /tmp/otelcol-metrics-zone-universe-main-dev.service /etc/systemd/system/
[2026-02-05 09:39:49 UTC] USER=www-data EUID=0 PID=576927 ACTION=passthru ARGS=mv /tmp/otelcol-metrics-zone-universe-main-dev.timer /etc/systemd/system/
[2026-02-05 09:39:49 UTC] USER=www-data EUID=0 PID=576936 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 09:39:49 UTC] USER=www-data EUID=0 PID=576993 ACTION=passthru ARGS=systemctl enable otelcol-metrics-zone-universe-main-dev.timer
Created symlink /etc/systemd/system/timers.target.wants/otelcol-metrics-zone-universe-main-dev.timer → /etc/systemd/system/otelcol-metrics-zone-universe-main-dev.timer.
[2026-02-05 09:39:50 UTC] USER=www-data EUID=0 PID=577083 ACTION=passthru ARGS=systemctl start otelcol-metrics-zone-universe-main-dev.timer
[OK]   ✅ Metrics collection timer installed and started
[OK]   Telemetry collector (otlp) deployed successfully

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Step 7/10: METRICS BACKEND DEPLOYMENT
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Provider: prometheus
[INFO]   OBS Cell: obs-zone-universe-main-dev
[INFO]   FQDN: metrics-zone-universe-main-dev-prometheus.fastorder.com
[INFO]   IP: 10.100.1.53
[INFO]   Script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/deploy-metrics.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📊 METRICS DEPLOYMENT WRAPPER STARTED
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Script: deploy-metrics.sh
[INFO] Timestamp: 2026-02-05 09:39:50 UTC
[INFO] Arguments: --provider prometheus --obs-cell obs-zone-universe-main-dev --fqdn metrics-zone-universe-main-dev-prometheus.fastorder.com --ip 10.100.1.53

[INFO] ═══════════════════════════════════════════════════════════════
[INFO] METRICS DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: prometheus
[INFO] Observability Cell: obs-zone-universe-main-dev
[INFO] FQDN: metrics-zone-universe-main-dev-prometheus.fastorder.com
[INFO] IP: 10.100.1.53
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: prometheus
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/provider/prometheus.sh

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/provider/prometheus.sh
[INFO]   OBS_CELL: obs-zone-universe-main-dev
[INFO]   FQDN: metrics-zone-universe-main-dev-prometheus.fastorder.com
[INFO]   IP: 10.100.1.53
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Parsed: SERVICE=zone, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-zone-universe-main-dev...
[2026-02-05 09:39:50 UTC] USER=www-data EUID=0 PID=577144 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-02-05 09:39:50 UTC] USER=www-data EUID=0 PID=577173 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-02-05 09:39:50 UTC] USER=www-data EUID=0 PID=577197 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-02-05 09:39:50 UTC] USER=www-data EUID=0 PID=577211 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-zone-universe-main-dev
[INFO] IP Address: 10.100.1.53
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-zone-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service clickhouse-server@obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server@obs-zone-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-zone-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-zone-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service prometheus-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Found 23 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.53...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[INFO] Deploying Prometheus for observability cell: obs-zone-universe-main-dev
[INFO] FQDN: metrics-zone-universe-main-dev-prometheus.fastorder.com
[INFO] IP: 10.100.1.53
[INFO] Prometheus Port: 9090
[VERSION] Fetching latest version for prometheus from GitHub (prometheus/prometheus)...
[VERSION] Latest prometheus version: 3.9.1
[INFO] Resolved Prometheus version: 3.9.1
[INFO] Checking if Prometheus is installed...
[OK]   Prometheus already installed at /usr/local/bin/prometheus
[VERSION] Fetching latest version for node_exporter from GitHub (prometheus/node_exporter)...
[VERSION] Latest node_exporter version: 1.10.2
[INFO] Resolved Node Exporter version: 1.10.2
[INFO] Checking if Node Exporter is installed...
[OK]   Node Exporter already installed at /usr/local/bin/node_exporter
[2026-02-05 09:39:51 UTC] USER=www-data EUID=0 PID=577522 ACTION=fsop ARGS=mkdir -p /etc/prometheus/obs-zone-universe-main-dev
[INFO] Creating Node Exporter TLS web config...
[INFO] Creating Node Exporter systemd service with TLS...
[2026-02-05 09:39:51 UTC] USER=www-data EUID=0 PID=577558 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 09:39:52 UTC] USER=www-data EUID=0 PID=577606 ACTION=passthru ARGS=systemctl enable node_exporter-obs-zone-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/node_exporter-obs-zone-universe-main-dev.service → /etc/systemd/system/node_exporter-obs-zone-universe-main-dev.service.
[2026-02-05 09:39:53 UTC] USER=www-data EUID=0 PID=577664 ACTION=passthru ARGS=systemctl restart node_exporter-obs-zone-universe-main-dev.service
[OK]   Node Exporter service configured and started
[INFO] Creating configuration directory: /etc/prometheus/obs-zone-universe-main-dev
[2026-02-05 09:39:53 UTC] USER=www-data EUID=0 PID=577680 ACTION=fsop ARGS=mkdir -p /etc/prometheus/obs-zone-universe-main-dev
[2026-02-05 09:39:53 UTC] USER=www-data EUID=0 PID=577689 ACTION=fsop ARGS=mkdir -p /var/lib/prometheus/obs-zone-universe-main-dev
[2026-02-05 09:39:53 UTC] USER=www-data EUID=0 PID=577698 ACTION=fsop ARGS=mkdir -p /etc/prometheus/obs-zone-universe-main-dev/rules
[INFO] Creating Prometheus configuration...
[INFO] Generated FQDNs:
[INFO]   Prometheus:   metrics-zone-universe-main-dev-prometheus.fastorder.com
[INFO]   Alertmanager: alerts-zone-universe-main-dev-alertmanager.fastorder.com
[INFO]   Grafana:      dashboards-zone-universe-main-dev-grafana.fastorder.com
[INFO]   Otelcol:      telemetry-zone-universe-main-dev-opentelemetry.fastorder.com
[OK]   Configuration created at /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml
[INFO] Creating Prometheus web config for HTTPS...
[OK]   Web config created at /etc/prometheus/obs-zone-universe-main-dev/web-config.yml
[INFO] Creating basic alerting rules...
[OK]   Alerting rules created
[2026-02-05 09:39:53 UTC] USER=www-data EUID=0 PID=577742 ACTION=fsop ARGS=mkdir -p /etc/prometheus/obs-zone-universe-main-dev/targets
[2026-02-05 09:39:53 UTC] USER=www-data EUID=0 PID=577756 ACTION=passthru ARGS=bash -c cat > '/etc/prometheus/obs-zone-universe-main-dev/targets/.placeholder.yml' << 'EOF'
# Placeholder file to prevent file_sd_configs warning
# Application targets will be added here automatically
[]
EOF
[INFO] Creating systemd service: prometheus-obs-zone-universe-main-dev
[INFO] Binding to: 10.100.1.53:9090
[OK]   Systemd service created
[INFO] Configuring certificate permissions...
[INFO] Configuring certificate permissions for prometheus (user: root)
[INFO] Initializing certificate directory for obs-zone-universe-main-dev...
[2026-02-05 09:39:53 UTC] USER=www-data EUID=0 PID=577778 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-02-05 09:39:53 UTC] USER=www-data EUID=0 PID=577787 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-02-05 09:39:53 UTC] USER=www-data EUID=0 PID=577796 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-02-05 09:39:53 UTC] USER=www-data EUID=0 PID=577805 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[INFO]   Setting file permissions...
[2026-02-05 09:39:53 UTC] USER=www-data EUID=0 PID=577815 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/prometheus-cert.pem
[2026-02-05 09:39:53 UTC] USER=www-data EUID=0 PID=577824 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem
[2026-02-05 09:39:53 UTC] USER=www-data EUID=0 PID=577833 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/prometheus-key.pem
[INFO]   Setting file ownership...
[2026-02-05 09:39:53 UTC] USER=www-data EUID=0 PID=577842 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-zone-universe-main-dev/prometheus-key.pem
[2026-02-05 09:39:53 UTC] USER=www-data EUID=0 PID=577851 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-zone-universe-main-dev/prometheus-cert.pem /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for prometheus
[OK]   Certificate permissions configured
[INFO] Adding /etc/hosts entry for metrics-zone-universe-main-dev-prometheus.fastorder.com -> 10.100.1.53
[2026-02-05 09:39:53 UTC] USER=www-data EUID=0 PID=577862 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*metrics-zone-universe-main-dev-prometheus.fastorder.com/10.100.1.53    metrics-zone-universe-main-dev-prometheus.fastorder.com/ /etc/hosts
[OK]   Updated /etc/hosts entry to use VM_IP
[INFO] Validating Prometheus configuration...
Checking /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml is valid prometheus config file syntax

Checking /etc/prometheus/obs-zone-universe-main-dev/rules/basic_alerts.yml
  SUCCESS: 4 rules found

[OK]   ✅ Configuration is valid
[INFO] Storing Prometheus configuration in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/zone/universe/main/dev/prometheus/server-AV5OjO",
    "Name": "fastorder/observability/zone/universe/main/dev/prometheus/server",
    "VersionId": "ab820dfa-b988-4b36-a71d-b15edb4a8f4e"
}
[OK]   Configuration stored in AWS Secrets Manager
[INFO] Enabling and starting Prometheus service...
[2026-02-05 09:39:55 UTC] USER=www-data EUID=0 PID=577911 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 09:39:56 UTC] USER=www-data EUID=0 PID=577960 ACTION=passthru ARGS=systemctl enable prometheus-obs-zone-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus-obs-zone-universe-main-dev.service → /etc/systemd/system/prometheus-obs-zone-universe-main-dev.service.
[2026-02-05 09:39:56 UTC] USER=www-data EUID=0 PID=578009 ACTION=passthru ARGS=systemctl restart prometheus-obs-zone-universe-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-02-05 09:39:59 UTC] USER=www-data EUID=0 PID=578123 ACTION=passthru ARGS=systemctl is-active --quiet prometheus-obs-zone-universe-main-dev.service
[OK]   ✅ Prometheus is running
[OK]   ✅ Prometheus web interface listening on port 9090
[OK]   ✅ Prometheus health check passed (HTTPS)
[INFO] ═══════════════════════════════════════════════════════════════
[OK]   Prometheus Web UI: https://metrics-zone-universe-main-dev-prometheus.fastorder.com:9090
[OK]   Targets: https://metrics-zone-universe-main-dev-prometheus.fastorder.com:9090/targets
[OK]   Alerts: https://metrics-zone-universe-main-dev-prometheus.fastorder.com:9090/alerts
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service logs (last 10 lines):
[2026-02-05 09:40:01 UTC] USER=www-data EUID=0 PID=578252 ACTION=passthru ARGS=journalctl -u prometheus-obs-zone-universe-main-dev.service -n 10 --no-pager
Feb 05 09:39:56 web-03 prometheus-obs-zone-universe-main-dev[578039]: ts=2026-02-05T09:39:56.612Z caller=head.go:722 level=info component=tsdb msg="Replaying WAL, this may take a while"
Feb 05 09:39:56 web-03 prometheus-obs-zone-universe-main-dev[578039]: ts=2026-02-05T09:39:56.613Z caller=head.go:794 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
Feb 05 09:39:56 web-03 prometheus-obs-zone-universe-main-dev[578039]: ts=2026-02-05T09:39:56.613Z caller=head.go:831 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=34.055µs wal_replay_duration=1.004313ms wbl_replay_duration=200ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=2.335µs total_replay_duration=1.065678ms
Feb 05 09:39:56 web-03 prometheus-obs-zone-universe-main-dev[578039]: ts=2026-02-05T09:39:56.615Z caller=main.go:1218 level=info fs_type=EXT4_SUPER_MAGIC
Feb 05 09:39:56 web-03 prometheus-obs-zone-universe-main-dev[578039]: ts=2026-02-05T09:39:56.616Z caller=main.go:1221 level=info msg="TSDB started"
Feb 05 09:39:56 web-03 prometheus-obs-zone-universe-main-dev[578039]: ts=2026-02-05T09:39:56.616Z caller=main.go:1404 level=info msg="Loading configuration file" filename=/etc/prometheus/obs-zone-universe-main-dev/prometheus.yml
Feb 05 09:39:56 web-03 prometheus-obs-zone-universe-main-dev[578039]: ts=2026-02-05T09:39:56.620Z caller=main.go:1441 level=info msg="updated GOGC" old=100 new=75
Feb 05 09:39:56 web-03 prometheus-obs-zone-universe-main-dev[578039]: ts=2026-02-05T09:39:56.620Z caller=main.go:1452 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/obs-zone-universe-main-dev/prometheus.yml totalDuration=4.397192ms db_storage=30.108µs remote_storage=2.415µs web_handler=752ns query_engine=1.814µs scrape=628.076µs scrape_sd=299.56µs notify=28.023µs notify_sd=10.08µs rules=1.988816ms tracing=6.642µs
Feb 05 09:39:56 web-03 prometheus-obs-zone-universe-main-dev[578039]: ts=2026-02-05T09:39:56.620Z caller=main.go:1182 level=info msg="Server is ready to receive web requests."
Feb 05 09:39:56 web-03 prometheus-obs-zone-universe-main-dev[578039]: ts=2026-02-05T09:39:56.621Z caller=manager.go:164 level=info component="rule manager" msg="Starting rule manager..."
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Provider script completed with exit code: 0
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Metrics Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: prometheus
[INFO] FQDN: metrics-zone-universe-main-dev-prometheus.fastorder.com
[INFO] IP: 10.100.1.53
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering Prometheus in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Prometheus
[INFO]   Identifier:        zone-universe-main-dev-prometheus
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.53
[INFO]   Port:              9090
[INFO]   FQDN:              metrics-zone-universe-main-dev-prometheus.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 9e9525a5-f460-4fe3-8eab-aeffb2dbeaa8
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   Prometheus registered in dashboard

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Step 7/10: METRICS DEPLOYMENT RESULT
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Exit code: 0
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ Metrics backend (prometheus) deployed successfully
[INFO] Step 8/10: Deploying traces backend...
[INFO]   Provider: tempo (selected)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] TRACES DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: tempo
[INFO] Observability Cell: obs-zone-universe-main-dev
[INFO] FQDN: traces-zone-universe-main-dev-tempo.fastorder.com
[INFO] IP: 10.100.1.200
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: tempo
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Traces/provider/tempo.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=zone, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-zone-universe-main-dev...
[2026-02-05 09:40:02 UTC] USER=www-data EUID=0 PID=578381 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-02-05 09:40:02 UTC] USER=www-data EUID=0 PID=578399 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-02-05 09:40:02 UTC] USER=www-data EUID=0 PID=578414 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-02-05 09:40:02 UTC] USER=www-data EUID=0 PID=578425 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-zone-universe-main-dev
[INFO] IP Address: 10.100.1.200
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-zone-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service clickhouse-server@obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server@obs-zone-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-zone-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-zone-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service prometheus-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-zone-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service grafana-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Found 24 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.200...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[INFO] Binding Tempo to allocated IP: 10.100.1.200
[INFO] Deploying Grafana Tempo for observability cell: obs-zone-universe-main-dev
[INFO] FQDN: traces-zone-universe-main-dev-tempo.fastorder.com
[INFO] Allocated IP: 10.100.1.200
[INFO] VM IP: 10.100.1.200
[INFO] Ports: HTTP=3200 gRPC=9295, OTLP gRPC=4317, OTLP HTTP=4318
[VERSION] Fetching latest version for tempo from GitHub (grafana/tempo)...
[VERSION] Latest tempo version: 2.10.0
[INFO] Resolved Tempo version: 2.10.0
[INFO] Checking if Grafana Tempo is installed...
[OK]   Grafana Tempo already installed at /usr/local/bin/tempo
[INFO] Preparing configuration and data directories...
[2026-02-05 09:40:04 UTC] USER=www-data EUID=0 PID=578708 ACTION=passthru ARGS=mkdir -p /etc/tempo/obs-zone-universe-main-dev
[2026-02-05 09:40:04 UTC] USER=www-data EUID=0 PID=578726 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-zone-universe-main-dev
[2026-02-05 09:40:04 UTC] USER=www-data EUID=0 PID=578738 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-zone-universe-main-dev/wal
[2026-02-05 09:40:04 UTC] USER=www-data EUID=0 PID=578748 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-zone-universe-main-dev/blocks
[2026-02-05 09:40:04 UTC] USER=www-data EUID=0 PID=578759 ACTION=passthru ARGS=chown -R tempo:tempo /etc/tempo/obs-zone-universe-main-dev /var/lib/tempo/obs-zone-universe-main-dev
[2026-02-05 09:40:04 UTC] USER=www-data EUID=0 PID=578771 ACTION=passthru ARGS=chmod 750 /etc/tempo/obs-zone-universe-main-dev /var/lib/tempo/obs-zone-universe-main-dev
[INFO] Creating Grafana Tempo configuration...
[INFO] TLS configuration exported for tempo
[INFO]   Cert: /etc/fastorder/observability/certs/obs-zone-universe-main-dev/tempo-cert.pem
[INFO]   Key:  /etc/fastorder/observability/certs/obs-zone-universe-main-dev/tempo-key.pem
[INFO]   CA:   /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem
[INFO] Setting up certificate permissions for Tempo...
[INFO] Configuring certificate permissions for tempo (user: tempo)
[INFO] Initializing certificate directory for obs-zone-universe-main-dev...
[2026-02-05 09:40:04 UTC] USER=www-data EUID=0 PID=578790 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-02-05 09:40:04 UTC] USER=www-data EUID=0 PID=578801 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-02-05 09:40:04 UTC] USER=www-data EUID=0 PID=578810 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-02-05 09:40:04 UTC] USER=www-data EUID=0 PID=578819 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[INFO]   Setting file permissions...
[2026-02-05 09:40:04 UTC] USER=www-data EUID=0 PID=578830 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/tempo-cert.pem
[2026-02-05 09:40:04 UTC] USER=www-data EUID=0 PID=578839 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem
[2026-02-05 09:40:04 UTC] USER=www-data EUID=0 PID=578848 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/tempo-key.pem
[INFO]   Setting file ownership...
[2026-02-05 09:40:04 UTC] USER=www-data EUID=0 PID=578858 ACTION=passthru ARGS=chown root:tempo /etc/fastorder/observability/certs/obs-zone-universe-main-dev/tempo-key.pem
[2026-02-05 09:40:04 UTC] USER=www-data EUID=0 PID=578867 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-zone-universe-main-dev/tempo-cert.pem /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for tempo
[2026-02-05 09:40:04 UTC] USER=www-data EUID=0 PID=578888 ACTION=passthru ARGS=chown tempo:tempo /etc/tempo/obs-zone-universe-main-dev/config.yaml
[2026-02-05 09:40:04 UTC] USER=www-data EUID=0 PID=578903 ACTION=passthru ARGS=chmod 640 /etc/tempo/obs-zone-universe-main-dev/config.yaml
[OK]   Configuration created at /etc/tempo/obs-zone-universe-main-dev/config.yaml
[INFO] Creating systemd service: tempo-obs-zone-universe-main-dev
[OK]   Systemd service created
[INFO] Adding /etc/hosts entry for traces-zone-universe-main-dev-tempo.fastorder.com -> 10.100.1.200
[2026-02-05 09:40:05 UTC] USER=www-data EUID=0 PID=578933 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*traces-zone-universe-main-dev-tempo.fastorder.com/10.100.1.200    traces-zone-universe-main-dev-tempo.fastorder.com/ /etc/hosts
[OK]   Updated /etc/hosts entry to use VM_IP
[INFO] Storing Tempo configuration in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/zone/universe/main/dev/tempo/server-XjGCwi",
    "Name": "fastorder/observability/zone/universe/main/dev/tempo/server",
    "VersionId": "57b7fd81-2850-46d0-a38a-336741fb4c1b"
}
[OK]   Tempo configuration stored/updated in AWS Secrets Manager: fastorder/observability/zone/universe/main/dev/tempo/server
[WARN] Port cleanup library not found, skipping automatic cleanup
[INFO] Adding iptables redirect for Tempo internal communication (required for search)...
[INFO] ╔════════════════════════════════════════════════════════════════════════╗
[INFO] ║  TEMPO IPTABLES DNAT CONFIGURATION (Audit Log)                        ║
[INFO] ╠════════════════════════════════════════════════════════════════════════╣
[INFO] ║  OBS_CELL:     obs-zone-universe-main-dev
[INFO] ║  VM_IP:        10.100.1.200
[INFO] ║  GRPC_PORT:    9295 (unique: 9095 + last_octet)
[INFO] ║  TEMPO_UID:    989
[INFO] ║  TIMESTAMP:    2026-02-05T09:40:07Z
[INFO] ╚════════════════════════════════════════════════════════════════════════╝
[INFO] Using --uid-owner 989 for DNAT rule (scoped to tempo user)
[2026-02-05 09:40:07 UTC] USER=www-data EUID=0 PID=579064 ACTION=passthru ARGS=iptables -t nat -A OUTPUT -p tcp -d 127.0.0.1 --dport 9295 -m owner --uid-owner 989 -j DNAT --to-destination 10.100.1.200:9295
ERROR: passthru not allowed: iptables
[ERR]  Could not add iptables redirect (iptables not allowed in wrapper)
[ERR]  ╔════════════════════════════════════════════════════════════════════════╗
[ERR]  ║  CRITICAL: Tempo search will NOT work without this redirect!           ║
[ERR]  ║                                                                        ║
[ERR]  ║  Root cause: Tempo single-binary dials 127.0.0.1:<grpc_port>          ║
[ERR]  ║  Each instance needs unique port + matching DNAT rule.                ║
[ERR]  ║                                                                        ║
[ERR]  ║  Manually run:                                                         ║
[ERR]  ║  sudo iptables -t nat -A OUTPUT -p tcp -d 127.0.0.1 --dport 9295 -m owner --uid-owner 989 \                   ║
[ERR]  ║    -j DNAT --to-destination 10.100.1.200:9295                                                  ║
[ERR]  ╚════════════════════════════════════════════════════════════════════════╝
[INFO] Enabling and starting Grafana Tempo service...
[2026-02-05 09:40:07 UTC] USER=www-data EUID=0 PID=579072 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 09:40:08 UTC] USER=www-data EUID=0 PID=579123 ACTION=passthru ARGS=systemctl enable tempo-obs-zone-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/tempo-obs-zone-universe-main-dev.service → /etc/systemd/system/tempo-obs-zone-universe-main-dev.service.
[2026-02-05 09:40:08 UTC] USER=www-data EUID=0 PID=579191 ACTION=passthru ARGS=systemctl restart tempo-obs-zone-universe-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-02-05 09:40:12 UTC] USER=www-data EUID=0 PID=579362 ACTION=passthru ARGS=systemctl is-active --quiet tempo-obs-zone-universe-main-dev.service
[OK]   ✅ Grafana Tempo is running
[OK]   ✅ HTTP endpoint listening on port 3200
[OK]   ✅ OTLP gRPC endpoint listening on port 4317
[OK]   ✅ OTLP HTTP endpoint listening on port 4318
[INFO] Running smoke test: Tempo search endpoint...
[WARN] ⚠️  Tempo search smoke test failed - check iptables DNAT rule
[WARN]     Expected JSON with completedJobs/totalJobs, got: <html>
    <head>
        <title>Page Not Found</title>
        <style>
            body{
          
[INFO] Service logs (last 10 lines):
[2026-02-05 09:40:14 UTC] USER=www-data EUID=0 PID=579402 ACTION=passthru ARGS=journalctl -u tempo-obs-zone-universe-main-dev.service -n 10 --no-pager
Feb 05 09:40:09 web-03 tempo-obs-zone-universe-main-dev[579241]: level=info ts=2026-02-05T09:40:09.575314936Z caller=compactor.go:140 msg="waiting until compactor ring topology is stable" min_waiting=1m0s max_waiting=5m0s
Feb 05 09:40:09 web-03 tempo-obs-zone-universe-main-dev[579241]: level=info ts=2026-02-05T09:40:09.572960173Z caller=worker.go:250 msg="total worker concurrency updated" totalConcurrency=20
Feb 05 09:40:09 web-03 tempo-obs-zone-universe-main-dev[579241]: ts=2026-02-05T09:40:09Z level=info msg="Starting GRPC server" component=tempo endpoint=10.100.1.200:4317
Feb 05 09:40:09 web-03 tempo-obs-zone-universe-main-dev[579241]: level=info ts=2026-02-05T09:40:09.57649932Z caller=lifecycler.go:714 msg="instance not found in ring, adding with no tokens" ring=ingester
Feb 05 09:40:09 web-03 tempo-obs-zone-universe-main-dev[579241]: level=info ts=2026-02-05T09:40:09.57667252Z caller=lifecycler.go:556 msg="auto-joining cluster after timeout" ring=ingester
Feb 05 09:40:09 web-03 tempo-obs-zone-universe-main-dev[579241]: ts=2026-02-05T09:40:09Z level=info msg="Starting HTTP server" component=tempo endpoint=10.100.1.200:4318
Feb 05 09:40:09 web-03 tempo-obs-zone-universe-main-dev[579241]: ts=2026-02-05T09:40:09Z level=info msg="Starting UDP server for Binary Thrift" component=tempo endpoint=10.100.1.200:7032
Feb 05 09:40:09 web-03 tempo-obs-zone-universe-main-dev[579241]: ts=2026-02-05T09:40:09Z level=info msg="Starting UDP server for Compact Thrift" component=tempo endpoint=10.100.1.200:7031
Feb 05 09:40:09 web-03 tempo-obs-zone-universe-main-dev[579241]: ts=2026-02-05T09:40:09Z level=info msg="Starting HTTP server for Jaeger Thrift" component=tempo endpoint=10.100.1.200:14468
Feb 05 09:40:09 web-03 tempo-obs-zone-universe-main-dev[579241]: ts=2026-02-05T09:40:09Z level=info msg="Starting gRPC server for Jaeger Protobuf" component=tempo endpoint=10.100.1.200:14450

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Traces Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: tempo
[INFO] FQDN: traces-zone-universe-main-dev-tempo.fastorder.com
[INFO] IP: 10.100.1.200
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering Tempo in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Tempo
[INFO]   Identifier:        zone-universe-main-dev-tempo
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.200
[INFO]   Port:              3200
[INFO]   FQDN:              traces-zone-universe-main-dev-tempo.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: a1080aad-2f0b-4249-ae48-33c815cf58d4
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   ✅ Tempo registered in dashboard
[OK]   Traces backend (tempo) deployed successfully
[INFO] Step 9/10: Deploying dashboards...
[INFO]   Provider: grafana (selected)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] DASHBOARDS DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: grafana
[INFO] Observability Cell: obs-zone-universe-main-dev
[INFO] FQDN: dashboards-zone-universe-main-dev-grafana.fastorder.com
[INFO] IP: 10.100.1.194
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: grafana
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Dashboards/provider/grafana.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=zone, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Binding to allocated IP: 10.100.1.194
[INFO] Deploying Grafana for observability cell: obs-zone-universe-main-dev
[INFO] FQDN: dashboards-zone-universe-main-dev-grafana.fastorder.com
[INFO] Allocated IP: 10.100.1.194
[INFO] VM IP: 10.100.1.194
[INFO] HTTP Port: 3000
[INFO] Checking if Grafana is installed...
[OK]   Grafana already installed
[INFO] Installing Grafana plugins...
[INFO] Installing ClickHouse datasource plugin...
[WARN] Failed to install ClickHouse plugin (may need internet access)
[INFO] Validating TLS certificate and key...
[INFO] Creating certificate symlinks...
[OK]   Certificate symlinks created
[INFO] Setting certificate permissions...
[OK]   TLS cert/key found and permissions set
[INFO] Creating configuration and data directories...
[2026-02-05 09:40:15 UTC] USER=www-data EUID=0 PID=579497 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-zone-universe-main-dev
[2026-02-05 09:40:15 UTC] USER=www-data EUID=0 PID=579506 ACTION=passthru ARGS=mkdir -p /var/lib/grafana/obs-zone-universe-main-dev
[2026-02-05 09:40:15 UTC] USER=www-data EUID=0 PID=579515 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-zone-universe-main-dev/provisioning/datasources
[2026-02-05 09:40:15 UTC] USER=www-data EUID=0 PID=579524 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-zone-universe-main-dev/provisioning/dashboards
[INFO] Creating Grafana configuration at /etc/grafana/obs-zone-universe-main-dev/grafana.ini...
[OK]   Configuration created
[INFO] Creating Prometheus datasource provisioning...
[OK]   Prometheus datasource provisioned
[INFO] Creating Tempo datasource provisioning...
[OK]   Tempo datasource provisioned
[INFO] Creating Loki datasource provisioning...
[OK]   Loki datasource provisioned
[INFO] Creating ClickHouse datasource provisioning...
[OK]   Retrieved ClickHouse credentials from Secrets Manager
[OK]   ClickHouse datasource provisioned
[INFO] Creating systemd service: grafana-obs-zone-universe-main-dev
[OK]   Systemd service created
[2026-02-05 09:40:17 UTC] USER=www-data EUID=0 PID=579673 ACTION=passthru ARGS=chown -R grafana:grafana /etc/grafana/obs-zone-universe-main-dev
[2026-02-05 09:40:17 UTC] USER=www-data EUID=0 PID=579691 ACTION=passthru ARGS=chmod 750 /etc/grafana/obs-zone-universe-main-dev /var/lib/grafana/obs-zone-universe-main-dev
[INFO] Adding /etc/hosts entry for dashboards-zone-universe-main-dev-grafana.fastorder.com -> 10.100.1.194
[WARN] /etc/hosts entry already exists
[INFO] Storing Grafana credentials in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/zone/universe/main/dev/grafana/admin-VyEsVP",
    "Name": "fastorder/observability/zone/universe/main/dev/grafana/admin",
    "VersionId": "2153c3a3-7187-4637-9ffd-987403778fed"
}
[OK]   Credentials stored in AWS Secrets Manager: fastorder/observability/zone/universe/main/dev/grafana/admin
[INFO] Enabling and starting Grafana service...
[2026-02-05 09:40:19 UTC] USER=www-data EUID=0 PID=579767 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 09:40:20 UTC] USER=www-data EUID=0 PID=579819 ACTION=passthru ARGS=systemctl enable grafana-obs-zone-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/grafana-obs-zone-universe-main-dev.service → /etc/systemd/system/grafana-obs-zone-universe-main-dev.service.
[2026-02-05 09:40:21 UTC] USER=www-data EUID=0 PID=579913 ACTION=passthru ARGS=systemctl restart grafana-obs-zone-universe-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-02-05 09:40:26 UTC] USER=www-data EUID=0 PID=580186 ACTION=passthru ARGS=systemctl is-active --quiet grafana-obs-zone-universe-main-dev.service
[OK]   ✅ Grafana is running
[OK]   ✅ Grafana web interface listening on port 3000
[INFO] ═══════════════════════════════════════════════════════════════
[OK]   Grafana Dashboard URL: https://dashboards-zone-universe-main-dev-grafana.fastorder.com:3000
[OK]   Username: admin
[OK]   Password is stored in AWS Secrets Manager at: fastorder/observability/zone/universe/main/dev/grafana/admin
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service logs (last 10 lines):
[2026-02-05 09:40:26 UTC] USER=www-data EUID=0 PID=580197 ACTION=passthru ARGS=journalctl -u grafana-obs-zone-universe-main-dev.service -n 10 --no-pager
Feb 05 09:40:26 web-03 grafana-obs-zone-universe-main-dev[579929]: logger=migrator t=2026-02-05T09:40:26.172538048Z level=info msg="Migration successfully executed" id="create data_keys table" duration=2.177626ms
Feb 05 09:40:26 web-03 grafana-obs-zone-universe-main-dev[579929]: logger=migrator t=2026-02-05T09:40:26.177183362Z level=info msg="Executing migration" id="create secrets table"
Feb 05 09:40:26 web-03 grafana-obs-zone-universe-main-dev[579929]: logger=migrator t=2026-02-05T09:40:26.178948242Z level=info msg="Migration successfully executed" id="create secrets table" duration=1.760902ms
Feb 05 09:40:26 web-03 grafana-obs-zone-universe-main-dev[579929]: logger=migrator t=2026-02-05T09:40:26.183408914Z level=info msg="Executing migration" id="rename data_keys name column to id"
Feb 05 09:40:26 web-03 grafana-obs-zone-universe-main-dev[579929]: logger=migrator t=2026-02-05T09:40:26.265141413Z level=info msg="Migration successfully executed" id="rename data_keys name column to id" duration=81.725205ms
Feb 05 09:40:26 web-03 grafana-obs-zone-universe-main-dev[579929]: logger=migrator t=2026-02-05T09:40:26.270273774Z level=info msg="Executing migration" id="add name column into data_keys"
Feb 05 09:40:26 web-03 grafana-obs-zone-universe-main-dev[579929]: logger=migrator t=2026-02-05T09:40:26.281355431Z level=info msg="Migration successfully executed" id="add name column into data_keys" duration=11.105361ms
Feb 05 09:40:26 web-03 grafana-obs-zone-universe-main-dev[579929]: logger=migrator t=2026-02-05T09:40:26.287457989Z level=info msg="Executing migration" id="copy data_keys id column values into name"
Feb 05 09:40:26 web-03 grafana-obs-zone-universe-main-dev[579929]: logger=migrator t=2026-02-05T09:40:26.287921772Z level=info msg="Migration successfully executed" id="copy data_keys id column values into name" duration=466.549µs
Feb 05 09:40:26 web-03 grafana-obs-zone-universe-main-dev[579929]: logger=migrator t=2026-02-05T09:40:26.293193298Z level=info msg="Executing migration" id="rename data_keys name column to label"

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Dashboards Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: grafana
[INFO] FQDN: dashboards-zone-universe-main-dev-grafana.fastorder.com
[INFO] IP: 10.100.1.194
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering Grafana in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Grafana
[INFO]   Identifier:        zone-universe-main-dev-grafana
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.194
[INFO]   Port:              3000
[INFO]   FQDN:              dashboards-zone-universe-main-dev-grafana.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 9290ea8e-ef9a-4f12-9300-15310d175721
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   ✅ Grafana registered in dashboard
[OK]   Dashboards (grafana) deployed successfully
[INFO] Step 10/10: Deploying alerting...
[INFO]   Provider: alertmanager (selected)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] ALERTING DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: alertmanager
[INFO] Observability Cell: obs-zone-universe-main-dev
[INFO] FQDN: alerts-zone-universe-main-dev-alertmanager.fastorder.com
[INFO] IP: 10.100.1.202
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Ports: Web=9093 Cluster=9094 (bound to IP: 10.100.1.202)

[INFO] Using provider: alertmanager
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Alerting/provider/alertmanager.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=zone, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Binding to allocated IP: 10.100.1.202
[INFO] Deploying Alertmanager for observability cell: obs-zone-universe-main-dev
[INFO] FQDN: alerts-zone-universe-main-dev-alertmanager.fastorder.com
[INFO] Allocated IP: 10.100.1.202
[INFO] VM IP: 10.100.1.202
[INFO] Ports: Web=9093 Cluster=9094
[VERSION] Fetching latest version for alertmanager from GitHub (prometheus/alertmanager)...
[VERSION] Latest alertmanager version: 0.31.0
[INFO] Resolved Alertmanager version: 0.31.0
[INFO] Checking if Alertmanager is installed...
[OK]   Alertmanager already installed at /usr/local/bin/alertmanager
[INFO] Validating TLS certificate and key...
[OK]   TLS cert/key found in /etc/fastorder/observability/certs/obs-zone-universe-main-dev
[INFO] Creating configuration and data directories...
[2026-02-05 09:40:27 UTC] USER=www-data EUID=0 PID=580322 ACTION=passthru ARGS=mkdir -p /etc/alertmanager/obs-zone-universe-main-dev
[2026-02-05 09:40:27 UTC] USER=www-data EUID=0 PID=580333 ACTION=passthru ARGS=mkdir -p /var/lib/alertmanager/obs-zone-universe-main-dev
[2026-02-05 09:40:27 UTC] USER=www-data EUID=0 PID=580342 ACTION=passthru ARGS=mkdir -p /etc/alertmanager/obs-zone-universe-main-dev/templates
[INFO] Creating Alertmanager configuration...
[OK]   Alertmanager configuration created at /etc/alertmanager/obs-zone-universe-main-dev/alertmanager.yml
[INFO] Creating notification templates...
[OK]   Notification templates created
[INFO] Creating Alertmanager web TLS configuration with mTLS...
[OK]   Web mTLS configuration created at /etc/alertmanager/obs-zone-universe-main-dev/web-config.yml
[INFO] Validating Alertmanager configuration...
[2026-02-05 09:40:28 UTC] USER=www-data EUID=0 PID=580391 ACTION=passthru ARGS=chmod 644 /etc/alertmanager/obs-zone-universe-main-dev/alertmanager.yml
Checking '/etc/alertmanager/obs-zone-universe-main-dev/alertmanager.yml'  SUCCESS
Found:
 - global config
 - route
 - 6 inhibit rules
 - 5 receivers
 - 1 templates
  SUCCESS

[OK]   ✅ Configuration is valid
[INFO] Creating systemd service: alertmanager-obs-zone-universe-main-dev
[OK]   Systemd service created
[2026-02-05 09:40:28 UTC] USER=www-data EUID=0 PID=580418 ACTION=passthru ARGS=chown alertmanager:alertmanager /etc/fastorder/observability/certs/obs-zone-universe-main-dev/alertmanager-key.pem
[2026-02-05 09:40:28 UTC] USER=www-data EUID=0 PID=580427 ACTION=passthru ARGS=chown alertmanager:alertmanager /etc/fastorder/observability/certs/obs-zone-universe-main-dev/alertmanager-cert.pem
[2026-02-05 09:40:28 UTC] USER=www-data EUID=0 PID=580436 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-zone-universe-main-dev/ca-cert.pem
[2026-02-05 09:40:28 UTC] USER=www-data EUID=0 PID=580446 ACTION=passthru ARGS=chown -R alertmanager:alertmanager /etc/alertmanager/obs-zone-universe-main-dev
[2026-02-05 09:40:28 UTC] USER=www-data EUID=0 PID=580456 ACTION=passthru ARGS=chown -R alertmanager:alertmanager /var/lib/alertmanager/obs-zone-universe-main-dev
[2026-02-05 09:40:28 UTC] USER=www-data EUID=0 PID=580466 ACTION=passthru ARGS=chmod 750 /etc/alertmanager/obs-zone-universe-main-dev /var/lib/alertmanager/obs-zone-universe-main-dev
[INFO] Adding /etc/hosts entry for alerts-zone-universe-main-dev-alertmanager.fastorder.com -> 10.100.1.202
[WARN] /etc/hosts entry already exists
[INFO] Storing Alertmanager configuration in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/zone/universe/main/dev/alertmanager/server-th3tVj",
    "Name": "fastorder/observability/zone/universe/main/dev/alertmanager/server",
    "VersionId": "558e40a9-6ac8-4f04-bfeb-9a5c44ab8d61"
}
[OK]   Configuration stored in AWS Secrets Manager: fastorder/observability/zone/universe/main/dev/alertmanager/server
[INFO] Enabling and starting Alertmanager service...
[2026-02-05 09:40:31 UTC] USER=www-data EUID=0 PID=580554 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 09:40:31 UTC] USER=www-data EUID=0 PID=580613 ACTION=passthru ARGS=systemctl enable alertmanager-obs-zone-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/alertmanager-obs-zone-universe-main-dev.service → /etc/systemd/system/alertmanager-obs-zone-universe-main-dev.service.
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-02-05 09:40:35 UTC] USER=www-data EUID=0 PID=580786 ACTION=passthru ARGS=systemctl is-active --quiet alertmanager-obs-zone-universe-main-dev.service
[OK]   ✅ Alertmanager is running
[OK]   ✅ Alertmanager HTTPS web interface listening on port 9093
[OK]   ✅ Alertmanager cluster port listening on port 9094
[WARN] ⚠️  Alertmanager health check not responding yet (HTTPS)
[INFO] ═══════════════════════════════════════════════════════════════
[OK]   Alertmanager Web UI: https://alerts-zone-universe-main-dev-alertmanager.fastorder.com:9093
[OK]   API Endpoint:        https://alerts-zone-universe-main-dev-alertmanager.fastorder.com:9093/api/v2
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service logs (last 10 lines):
[2026-02-05 09:40:35 UTC] USER=www-data EUID=0 PID=580800 ACTION=passthru ARGS=journalctl -u alertmanager-obs-zone-universe-main-dev.service -n 10 --no-pager

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Alerting Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: alertmanager
[INFO] FQDN: alerts-zone-universe-main-dev-alertmanager.fastorder.com
[INFO] IP: 10.100.1.202
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Setting up HTTPS reverse proxy...
[INFO] Backend port: 9093
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Alertmanager HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-zone-universe-main-dev
  FQDN:         alerts-zone-universe-main-dev-alertmanager.fastorder.com
  Backend:      https://alerts-zone-universe-main-dev-alertmanager.fastorder.com:9093/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.202
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[ERROR] This script must be run as root or with sudo
[WARN] ⚠️  HTTPS setup failed (Alertmanager is still running on HTTP)
[INFO] Registering Alertmanager in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Alertmanager
[INFO]   Identifier:        zone-universe-main-dev-alertmanager
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.202
[INFO]   Port:              9093
[INFO]   FQDN:              alerts-zone-universe-main-dev-alertmanager.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: c81de39b-c158-44e1-bf29-e1f32c95481c
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   ✅ Alertmanager registered in dashboard
[OK]   Alerting (alertmanager) deployed successfully
[INFO] Step 10.5: Deploying Blackbox Exporter for synthetic monitoring...
[VERSION] Fetching latest version for blackbox_exporter from GitHub (prometheus/blackbox_exporter)...
[VERSION] Latest blackbox_exporter version: 0.28.0
[BLACKBOX] Resolved Blackbox Exporter version: 0.28.0
[BLACKBOX] Starting Blackbox Exporter deployment for obs-zone-universe-main-dev
[BLACKBOX] VM IP: 10.100.1.53
[BLACKBOX] Version: 0.28.0
[BLACKBOX] Checking prerequisites...
[BLACKBOX] Creating directories...
[BLACKBOX] Downloading Blackbox Exporter v0.28.0...
Sorry, user www-data is not allowed to execute '/usr/bin/mv /tmp/tmp.L1dhcjAJ14/blackbox_exporter-0.28.0.linux-amd64/blackbox_exporter /usr/local/bin/' as root on web-03.
[WARN] Blackbox Exporter deployment failed (non-fatal, synthetic monitoring disabled)
[INFO] Step 11/13: Configuring HTTPS reverse proxies...
[INFO] Setting up Prometheus HTTPS proxy...
[2026-02-05 09:40:38 UTC] USER=www-data EUID=0 PID=580987 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/https/setup-prometheus-https.sh --obs-cell obs-zone-universe-main-dev --backend-ip 10.100.1.53
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Prometheus HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-zone-universe-main-dev
  FQDN:         metrics-zone-universe-main-dev-prometheus.fastorder.com
  Backend:      https://metrics-zone-universe-main-dev-prometheus.fastorder.com:9090/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.53
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity (will retry up to 60s)...
[OK] Backend is accessible
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for metrics-zone-universe-main-dev-prometheus.fastorder.com

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/metrics-zone-universe-main-dev-prometheus.fastorder.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/metrics-zone-universe-main-dev-prometheus.fastorder.com/privkey.pem
This certificate expires on 2026-05-06.
These files will be updated when the certificate renews.
NEXT STEPS:
- The certificate will need to be renewed before it expires. Certbot can automatically renew the certificate in the background, but you may need to take steps to enable that functionality. See https://certbot.org/renewal-setup for instructions.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[OK] Certificate obtained
[INFO] Creating HTTPS VirtualHost with reverse proxy...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Setting up certificate auto-renewal...
[OK] Auto-renewal configured
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated
[INFO] Verifying HTTPS setup...
[OK] HTTPS endpoint is working

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Prometheus HTTPS Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HTTPS Endpoints:
  Health:   https://metrics-zone-universe-main-dev-prometheus.fastorder.com/-/healthy
  Ready:    https://metrics-zone-universe-main-dev-prometheus.fastorder.com/-/ready
  Graph:    https://metrics-zone-universe-main-dev-prometheus.fastorder.com/graph
  Targets:  https://metrics-zone-universe-main-dev-prometheus.fastorder.com/targets
  Alerts:   https://metrics-zone-universe-main-dev-prometheus.fastorder.com/alerts
  API:      https://metrics-zone-universe-main-dev-prometheus.fastorder.com/api/v1/...

Apache VirtualHosts:
  HTTP:  /etc/apache2/sites-available/metrics-zone-universe-main-dev-prometheus.fastorder.com.conf
  HTTPS: /etc/apache2/sites-available/metrics-zone-universe-main-dev-prometheus.fastorder.com-ssl.conf

Certificate:
  Path:     /etc/letsencrypt/live/metrics-zone-universe-main-dev-prometheus.fastorder.com/
  Renewal:  certbot renew --cert-name metrics-zone-universe-main-dev-prometheus.fastorder.com

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   Prometheus HTTPS proxy configured
[INFO] Setting up Grafana HTTPS proxy...
[2026-02-05 09:40:51 UTC] USER=www-data EUID=0 PID=581596 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Dashboards/https/setup-grafana-https.sh --obs-cell obs-zone-universe-main-dev --backend-ip 10.100.1.194
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-zone-universe-main-dev
  FQDN:         dashboards-zone-universe-main-dev-grafana.fastorder.com
  Backend:      https://dashboards-zone-universe-main-dev-grafana.fastorder.com:3000/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.194
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for dashboards-zone-universe-main-dev-grafana.fastorder.com

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/dashboards-zone-universe-main-dev-grafana.fastorder.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/dashboards-zone-universe-main-dev-grafana.fastorder.com/privkey.pem
This certificate expires on 2026-05-06.
These files will be updated when the certificate renews.
NEXT STEPS:
- The certificate will need to be renewed before it expires. Certbot can automatically renew the certificate in the background, but you may need to take steps to enable that functionality. See https://certbot.org/renewal-setup for instructions.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[OK] Certificate obtained
[INFO] Creating HTTPS VirtualHost...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana HTTPS Setup Complete!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  Grafana URL: https://dashboards-zone-universe-main-dev-grafana.fastorder.com/
  Metrics:     https://dashboards-zone-universe-main-dev-grafana.fastorder.com/metrics

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   Grafana HTTPS proxy configured
[INFO] Setting up OpenTelemetry Collector HTTPS proxy...
[2026-02-05 09:41:02 UTC] USER=www-data EUID=0 PID=582135 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Telemetry/https/setup-otelcol-https.sh --obs-cell obs-zone-universe-main-dev --backend-ip 10.100.1.203
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OpenTelemetry Collector HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-zone-universe-main-dev
  FQDN:         telemetry-zone-universe-main-dev-opentelemetry.fastorder.com
  Backend:      http://telemetry-zone-universe-main-dev-opentelemetry.fastorder.com:8888/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.203
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[OK] Backend is accessible and returning metrics via HTTPS
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for telemetry-zone-universe-main-dev-opentelemetry.fastorder.com

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/telemetry-zone-universe-main-dev-opentelemetry.fastorder.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/telemetry-zone-universe-main-dev-opentelemetry.fastorder.com/privkey.pem
This certificate expires on 2026-05-06.
These files will be updated when the certificate renews.
NEXT STEPS:
- The certificate will need to be renewed before it expires. Certbot can automatically renew the certificate in the background, but you may need to take steps to enable that functionality. See https://certbot.org/renewal-setup for instructions.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[OK] Certificate obtained
[INFO] Creating HTTPS VirtualHost with reverse proxy...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Setting up certificate auto-renewal...
[OK] Auto-renewal configured
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated
[INFO] Verifying HTTPS setup...
[OK] HTTPS endpoint is working and returning metrics

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OpenTelemetry Collector HTTPS Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HTTPS Endpoints:
  Metrics: https://telemetry-zone-universe-main-dev-opentelemetry.fastorder.com/metrics

Apache VirtualHosts:
  HTTP:  /etc/apache2/sites-available/telemetry-zone-universe-main-dev-opentelemetry.fastorder.com.conf
  HTTPS: /etc/apache2/sites-available/telemetry-zone-universe-main-dev-opentelemetry.fastorder.com-ssl.conf

Certificate:
  Path:     /etc/letsencrypt/live/telemetry-zone-universe-main-dev-opentelemetry.fastorder.com/
  Renewal:  certbot renew --cert-name telemetry-zone-universe-main-dev-opentelemetry.fastorder.com

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   OpenTelemetry Collector HTTPS proxy configured
[INFO] Setting up ClickHouse HTTPS proxy...
[2026-02-05 09:41:15 UTC] USER=www-data EUID=0 PID=582699 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/https/setup-clickhouse-https.sh --obs-cell obs-zone-universe-main-dev --backend-ip 10.100.1.199
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ClickHouse HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-zone-universe-main-dev
  FQDN:         logstore-zone-universe-main-dev.fastorder.com
  Backend:      http://logstore-zone-universe-main-dev.fastorder.com:8123/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.199
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity (will retry up to 60s)...
[OK] Backend is accessible
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for logstore-zone-universe-main-dev.fastorder.com

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/logstore-zone-universe-main-dev.fastorder.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/logstore-zone-universe-main-dev.fastorder.com/privkey.pem
This certificate expires on 2026-05-06.
These files will be updated when the certificate renews.
NEXT STEPS:
- The certificate will need to be renewed before it expires. Certbot can automatically renew the certificate in the background, but you may need to take steps to enable that functionality. See https://certbot.org/renewal-setup for instructions.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[OK] Certificate obtained
[INFO] Creating HTTPS VirtualHost with reverse proxy...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Setting up certificate auto-renewal...
[OK] Auto-renewal configured
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated
[INFO] Verifying HTTPS setup...
[OK] HTTPS endpoint is working

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ ClickHouse HTTPS Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HTTPS Endpoints:
  Health:     https://logstore-zone-universe-main-dev.fastorder.com/
  Dashboard:  https://logstore-zone-universe-main-dev.fastorder.com/dashboard
  Playground: https://logstore-zone-universe-main-dev.fastorder.com/play
  Metrics:    https://logstore-zone-universe-main-dev.fastorder.com/metrics

Login Instructions:
  1. Get credentials from skeleton: POST /api/monitoring/clickhouse/credentials
  2. Use auto-login URL: https://logstore-zone-universe-main-dev.fastorder.com/dashboard#user=<USER>&password=<PASS>
  3. Or use skeleton monitoring dashboard for one-click access

Apache VirtualHosts:
  HTTP:  /etc/apache2/sites-available/logstore-zone-universe-main-dev.fastorder.com.conf
  HTTPS: /etc/apache2/sites-available/logstore-zone-universe-main-dev.fastorder.com-ssl.conf

Certificate:
  Path: /etc/letsencrypt/live/logstore-zone-universe-main-dev.fastorder.com/
  Auto-renewal: Enabled via certbot.timer

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ClickHouse HTTPS proxy configured
[INFO] Setting up Tempo HTTPS proxy...
[2026-02-05 09:41:29 UTC] USER=www-data EUID=0 PID=583325 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Traces/https/setup-tempo-https.sh --obs-cell obs-zone-universe-main-dev --backend-ip 10.100.1.200
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana Tempo HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-zone-universe-main-dev
  FQDN:         traces-zone-universe-main-dev-tempo.fastorder.com
  Backend:      https://10.100.1.200:3200/
  Backend IP:   10.100.1.200
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[WARN] Cannot verify Tempo health endpoint (it may not be running yet), continuing anyway...
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for traces-zone-universe-main-dev-tempo.fastorder.com

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/traces-zone-universe-main-dev-tempo.fastorder.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/traces-zone-universe-main-dev-tempo.fastorder.com/privkey.pem
This certificate expires on 2026-05-06.
These files will be updated when the certificate renews.
NEXT STEPS:
- The certificate will need to be renewed before it expires. Certbot can automatically renew the certificate in the background, but you may need to take steps to enable that functionality. See https://certbot.org/renewal-setup for instructions.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[OK] Certificate obtained
[INFO] Generating Apache client certificate for mTLS backend connection...
[OK] Apache client certificate already exists
[INFO] Creating HTTPS VirtualHost with mTLS backend...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana Tempo HTTPS Setup Complete!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  Tempo URL:    https://traces-zone-universe-main-dev-tempo.fastorder.com/
  Ready:        https://traces-zone-universe-main-dev-tempo.fastorder.com/ready
  Metrics:      https://traces-zone-universe-main-dev-tempo.fastorder.com/metrics
  Build Info:   https://traces-zone-universe-main-dev-tempo.fastorder.com/api/status/buildinfo

  Note: Tempo backend must be running at traces-zone-universe-main-dev-tempo.fastorder.com:3200 (10.100.1.200)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   Tempo HTTPS proxy configured
[INFO] Setting up Alertmanager HTTPS proxy...
[2026-02-05 09:41:42 UTC] USER=www-data EUID=0 PID=583846 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Alerting/https/setup-alertmanager-https.sh --obs-cell obs-zone-universe-main-dev --backend-ip 10.100.1.202
[INFO] Backend port: 9093
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Alertmanager HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-zone-universe-main-dev
  FQDN:         alerts-zone-universe-main-dev-alertmanager.fastorder.com
  Backend:      https://alerts-zone-universe-main-dev-alertmanager.fastorder.com:9093/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.202
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[WARN] Backend health check inconclusive - proceeding anyway
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for alerts-zone-universe-main-dev-alertmanager.fastorder.com

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/alerts-zone-universe-main-dev-alertmanager.fastorder.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/alerts-zone-universe-main-dev-alertmanager.fastorder.com/privkey.pem
This certificate expires on 2026-05-06.
These files will be updated when the certificate renews.
NEXT STEPS:
- The certificate will need to be renewed before it expires. Certbot can automatically renew the certificate in the background, but you may need to take steps to enable that functionality. See https://certbot.org/renewal-setup for instructions.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[OK] Certificate obtained
[INFO] Creating HTTPS VirtualHost with reverse proxy...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Setting up certificate auto-renewal...
[OK] Auto-renewal configured
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated
[INFO] Verifying HTTPS setup...
OK[OK] HTTPS endpoint is working

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Alertmanager HTTPS Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HTTPS Endpoints:
  Health:   https://alerts-zone-universe-main-dev-alertmanager.fastorder.com/-/healthy
  Ready:    https://alerts-zone-universe-main-dev-alertmanager.fastorder.com/-/ready
  Web UI:   https://alerts-zone-universe-main-dev-alertmanager.fastorder.com/
  API:      https://alerts-zone-universe-main-dev-alertmanager.fastorder.com/api/v2/...

Apache VirtualHosts:
  HTTP:  /etc/apache2/sites-available/alerts-zone-universe-main-dev-alertmanager.fastorder.com.conf
  HTTPS: /etc/apache2/sites-available/alerts-zone-universe-main-dev-alertmanager.fastorder.com-ssl.conf

Certificate:
  Path:     /etc/letsencrypt/live/alerts-zone-universe-main-dev-alertmanager.fastorder.com/
  Renewal:  certbot renew --cert-name alerts-zone-universe-main-dev-alertmanager.fastorder.com

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   Alertmanager HTTPS proxy configured
[OK]   HTTPS reverse proxies configured
[INFO] Step 12/13: Configuring firewall rules (network segmentation)...

[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING FIREWALL RULES FOR OBSERVABILITY CELL
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Cell ID: obs-zone-universe-main-dev
[INFO] Internal Network: 10.0.0.0/8

[INFO] Discovering dashboard/skeleton VM IPs...
[INFO]   Discovered skeleton IP: 142.93.238.16 (skeleton.fastorder.com)
[INFO] Authorized dashboard IPs:
[INFO]   - 10.100.60.2
[INFO]   - 142.93.238.16

[INFO] Configuring UFW firewall rules...
[2026-02-05 09:41:54 UTC] USER=www-data EUID=0 PID=584548 ACTION=passthru ARGS=ufw default deny incoming
ERROR: passthru not allowed: ufw
[2026-02-05 09:41:54 UTC] USER=www-data EUID=0 PID=584558 ACTION=passthru ARGS=ufw default allow outgoing
ERROR: passthru not allowed: ufw
[2026-02-05 09:41:55 UTC] USER=www-data EUID=0 PID=584566 ACTION=passthru ARGS=ufw allow 22/tcp comment SSH
ERROR: passthru not allowed: ufw
[INFO]   Allowing prometheus (port 9090) from internal network...
[INFO]   Allowing alertmanager (port 9093) from internal network...
[INFO]   Allowing clickhouse (port 8123) from internal network...
[INFO]   Allowing grafana (port 3000) from internal network...
[INFO]   Allowing otelcol (port 4318) from internal network...
[INFO]   Allowing loki (port 3100) from internal network...
[INFO]   Allowing tempo (port 3200) from internal network...
[INFO]   Allowing dashboard access from 10.100.60.2...
[2026-02-05 09:41:55 UTC] USER=www-data EUID=0 PID=584638 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 9093 proto tcp comment Dashboard: alertmanager
ERROR: passthru not allowed: ufw
[2026-02-05 09:41:55 UTC] USER=www-data EUID=0 PID=584646 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 8123 proto tcp comment Dashboard: clickhouse
ERROR: passthru not allowed: ufw
[2026-02-05 09:41:55 UTC] USER=www-data EUID=0 PID=584654 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3000 proto tcp comment Dashboard: grafana
[2026-02-05 09:41:55 UTC] USER=www-data EUID=0 PID=584670 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 4318 proto tcp comment Dashboard: otelcol
ERROR: passthru not allowed: ufw
[2026-02-05 09:41:55 UTC] USER=www-data EUID=0 PID=584678 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3100 proto tcp comment Dashboard: loki
ERROR: passthru not allowed: ufw
[2026-02-05 09:41:55 UTC] USER=www-data EUID=0 PID=584688 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3200 proto tcp comment Dashboard: tempo
ERROR: passthru not allowed: ufw
[INFO]   Allowing dashboard access from 142.93.238.16...
[2026-02-05 09:41:55 UTC] USER=www-data EUID=0 PID=584720 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 3000 proto tcp comment Dashboard: grafana
ERROR: passthru not allowed: ufw
[2026-02-05 09:41:55 UTC] USER=www-data EUID=0 PID=584773 ACTION=passthru ARGS=ufw allow 443/tcp comment HTTPS obs-proxy
ERROR: passthru not allowed: ufw
[2026-02-05 09:41:55 UTC] USER=www-data EUID=0 PID=584789 ACTION=passthru ARGS=ufw reload
ERROR: passthru not allowed: ufw
[OK]   UFW firewall rules configured

[OK]   ═══════════════════════════════════════════════════════════════
[OK]   ✅ Firewall configuration completed
[OK]   ═══════════════════════════════════════════════════════════════

[INFO] Current firewall status:
[OK]   Firewall rules configured
[INFO] Step 13/13: Configuring OAuth/SSO...
[INFO] OAuth/SSO configuration script not found, skipping...

[INFO] Running validation checks...
[INFO] Validation script not found, skipping...

[INFO] Registering observability components to dashboard...
[INFO] Components to register: metrics alerts dashboards traces telemetry logstore proxy
[INFO]   Skipping metrics - registered by deploy script
[INFO]   Skipping alerts - registered by deploy script
[INFO]   Skipping dashboards - registered by deploy script
[INFO]   Skipping traces - registered by deploy script
[INFO]   Skipping telemetry - registered by deploy script
[INFO]   Skipping logstore - registered by deploy script
[INFO]   Processing component: proxy
[INFO] Registering: proxy (obs-zone-universe-main-dev-proxy)
[INFO] Detected observability component, parsing: zone-universe-main-dev-proxy
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Observability Proxy
[INFO]   Identifier:        obs-zone-universe-main-dev-proxy
[INFO]   Identifier Parent: observability-cell
[INFO]   IP:                10.100.1.73
[INFO]   Port:              443
[INFO]   FQDN:              observe-zone-universe-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 1794e8d7-3509-46fe-bca4-8a81f13fd0fa
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   ✓ Registered: proxy
[INFO] Registering short DNS aliases...
[OK]   ✓ Observability components registration completed

[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] Verifying all observability services are running...
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[OK]     ✓ grafana-obs-zone-universe-main-dev.service is running
[OK]     ✓ prometheus-obs-zone-universe-main-dev.service is running
[OK]     ✓ alertmanager-obs-zone-universe-main-dev.service is running
[OK]     ✓ otelcol-obs-zone-universe-main-dev.service is running
[OK]   ✓ All observability services verified running


═══════════════════════════════════════════════════════════════════════════════
[OK]   ✅ OBSERVABILITY CELL PROVISIONED: obs-zone-universe-main-dev
═══════════════════════════════════════════════════════════════════════════════

[INFO] DNS Entries:
  metrics-zone-universe-main-dev-prometheus.fastorder.com (10.100.1.53)
  alerts-zone-universe-main-dev-alertmanager.fastorder.com (10.100.1.202)
  dashboards-zone-universe-main-dev-grafana.fastorder.com (10.100.1.194)
  traces-zone-universe-main-dev-tempo.fastorder.com (10.100.1.200)
  telemetry-zone-universe-main-dev-opentelemetry.fastorder.com (10.100.1.203)
  logstore-zone-universe-main-dev-clickhouse.fastorder.com (10.100.1.199)
  observe-zone-universe-main-dev.fastorder.com (10.100.1.73)

[INFO] Secrets Path: fastorder/observability/zone/universe/dev/*

[INFO] Access (Purpose-Oriented URLs):
  Dashboards: https://dashboards-zone-universe-main-dev-grafana.fastorder.com (SSO enabled)
  Metrics: https://metrics-zone-universe-main-dev-prometheus.fastorder.com (internal only)
  Alerts: https://alerts-zone-universe-main-dev-alertmanager.fastorder.com
  Log Storage: https://logstore-zone-universe-main-dev-clickhouse.fastorder.com

[INFO] Backend Implementation (Internal - Not Exposed to Clients):
  Telemetry: otlp
  Metrics: prometheus
  Traces: tempo
  Dashboards: grafana
  Alerting: alertmanager
  Log Storage: clickhouse

[INFO] For applications in zone-universe-main-dev:
  - Metrics: Push to telemetry-zone-universe-main-dev-opentelemetry.fastorder.com:4318 (OTLP/HTTP)
  - Logs: Push to telemetry-zone-universe-main-dev-opentelemetry.fastorder.com:4318 (OTLP/HTTP)
  - Traces: Push to telemetry-zone-universe-main-dev-opentelemetry.fastorder.com:4317 (OTLP/gRPC)
  - Query Metrics: https://metrics-zone-universe-main-dev-prometheus.fastorder.com
  - Query Logs: https://logstore-zone-universe-main-dev-clickhouse.fastorder.com
  - Query Traces: https://traces-zone-universe-main-dev-tempo.fastorder.com

[INFO] Runbook: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/RUNBOOK.md
═══════════════════════════════════════════════════════════════════════════════

[INFO] Using search engine from SEARCH_ENGINE environment variable: elasticsearch
[INFO] Cleaning up any existing locks...

Starting search engine: elasticsearch
═══════════════════════════════════════════════

════════════════════════════════════════════════════════════════
           Elasticsearch Deployment Runner                        
════════════════════════════════════════════════════════════════

[INFO] Cleaning up any existing locks (without triggering package configurations)...
[WARNING] Lock cleanup skipped (wrapper not available or insufficient permissions)

🚀 Auto mode enabled - running automatic installation


Starting Automatic Installation...
═══════════════════════════════════════════════
Will execute all deployment tasks in sequence:

  [1] Install Elasticsearch Http (01-install-elasticsearch-http)
  [2] Make Https (02-make-https)
  [3] Create Index Llm (03-create-index-llm)
  [4] Monitoring Setup (10-monitoring-setup)

═══════════════════════════════════════════════
🚀 Auto mode - proceeding automatically...

Running automatic installation...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 1: Executing Install Elasticsearch Http
Folder: 01-install-elasticsearch-http
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

=== Elasticsearch HTTP Setup ===
Install and configure Elasticsearch with HTTP access
Architecture: Per-node VM IPs with default port (9200)

[INFO] Using web-provided environment: zone-universe-main-dev
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: 
Nodes: 1
Port: 9200 (default Elasticsearch port)
Coordinator endpoint: http://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

Checking if Elasticsearch is already installed for environment: ...
Validating Elasticsearch installation...
./run.sh: line 132: /var/www/html/skeleton.dev.fastorder.com/fixing/scripts/lib/elasticsearch_validator.sh: No such file or directory
⚠️  Elasticsearch installation issues detected. Attempting automatic repair...
./run.sh: line 134: /var/www/html/skeleton.dev.fastorder.com/fixing/scripts/lib/elasticsearch_validator.sh: No such file or directory
Executing: steps/01-setup-directories.sh
+ 01-setup-directories.sh:4:main: echo '=== Step 1: Creating directory structure ==='
=== Step 1: Creating directory structure ===
+++ 01-setup-directories.sh:4:main: dirname steps/01-setup-directories.sh
++ 01-setup-directories.sh:4:main: cd steps
++ 01-setup-directories.sh:4:main: pwd
+ 01-setup-directories.sh:4:main: SCRIPT_DIR=/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps
+ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/lib/provisioning-init.sh
++ 01-setup-directories.sh:4:main: RED='\033[0;31m'
++ 01-setup-directories.sh:4:main: GREEN='\033[0;32m'
++ 01-setup-directories.sh:4:main: YELLOW='\033[1;33m'
++ 01-setup-directories.sh:4:main: BLUE='\033[0;34m'
++ 01-setup-directories.sh:4:main: NC='\033[0m'
++ 01-setup-directories.sh:4:main: export TERM=dumb
++ 01-setup-directories.sh:4:main: TERM=dumb
++ 01-setup-directories.sh:4:main: export DEBIAN_FRONTEND=noninteractive
++ 01-setup-directories.sh:4:main: DEBIAN_FRONTEND=noninteractive
++ 01-setup-directories.sh:4:main: export NEEDRESTART_MODE=a
++ 01-setup-directories.sh:4:main: NEEDRESTART_MODE=a
++ 01-setup-directories.sh:4:main: export NEEDRESTART_SUSPEND=1
++ 01-setup-directories.sh:4:main: NEEDRESTART_SUSPEND=1
++ 01-setup-directories.sh:4:main: export DEBIAN_PRIORITY=critical
++ 01-setup-directories.sh:4:main: DEBIAN_PRIORITY=critical
++ 01-setup-directories.sh:4:main: export UCF_FORCE_CONFFOLD=1
++ 01-setup-directories.sh:4:main: UCF_FORCE_CONFFOLD=1
++ 01-setup-directories.sh:4:main: export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
++ 01-setup-directories.sh:4:main: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
++ 01-setup-directories.sh:4:main: [[ -n '' ]]
++ 01-setup-directories.sh:4:main: [[ -n /opt/fastorder/bash/scripts/env_app_setup/state ]]
++ 01-setup-directories.sh:4:main: [[ -d /opt/fastorder/bash/scripts/env_app_setup/state ]]
++ 01-setup-directories.sh:4:main: STATE_DIR=/opt/fastorder/bash/scripts/env_app_setup/state
++ 01-setup-directories.sh:4:main: export STATE_DIR
++ 01-setup-directories.sh:4:main: [[ -f /opt/fastorder/bash/scripts/env_app_setup/setup/setup.json ]]
++ 01-setup-directories.sh:4:main: SETUP_JSON=/opt/fastorder/bash/scripts/env_app_setup/setup/setup.json
++ 01-setup-directories.sh:4:main: FO_WRAPPER=/usr/local/bin/fastorder-provisioning-wrapper.sh
++ 01-setup-directories.sh:4:main: HTTP_PORT_BASE=9200
++ 01-setup-directories.sh:4:main: TRANSPORT_PORT_BASE=9300
++ 01-setup-directories.sh:4:main: PG_PORT_BASE=5432
++ 01-setup-directories.sh:4:main: APP_IP_SUBNETS=(['observability']='10.100.5' ['obs']='10.100.5' ['prometheus']='10.100.5' ['grafana']='10.100.5' ['loki']='10.100.5' ['tempo']='10.100.5' ['postgresql']='10.100.10' ['postgres']='10.100.10' ['pg']='10.100.10' ['elasticsearch']='10.100.20' ['es']='10.100.20' ['kafka']='10.100.30' ['redis']='10.100.40' ['mongodb']='10.100.50' ['mongo']='10.100.50' ['iam']='10.100.60' ['keycloak']='10.100.60' ['general']='10.100.1')
++ 01-setup-directories.sh:4:main: declare -A APP_IP_SUBNETS
++ 01-setup-directories.sh:4:main: APP_IP_RESERVED_START=(['observability']='2' ['postgresql']='2' ['elasticsearch']='2' ['kafka']='2' ['redis']='2' ['mongodb']='2' ['iam']='2' ['general']='50')
++ 01-setup-directories.sh:4:main: declare -A APP_IP_RESERVED_START
++ 01-setup-directories.sh:4:main: APP_IP_RESERVED_END=(['observability']='49' ['postgresql']='254' ['elasticsearch']='254' ['kafka']='254' ['redis']='254' ['mongodb']='254' ['iam']='254' ['general']='250')
++ 01-setup-directories.sh:4:main: declare -A APP_IP_RESERVED_END
+++ 01-setup-directories.sh:4:main: dirname /opt/fastorder/bash/scripts/env_app_setup/lib/provisioning-init.sh
++ 01-setup-directories.sh:4:main: _CONFIG_MGMT_LIB=/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
++ 01-setup-directories.sh:4:main: [[ -f /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh ]]
++ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
+++ 01-setup-directories.sh:4:main: set -Eeuo pipefail
+++ 01-setup-directories.sh:4:main: : /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
+++ 01-setup-directories.sh:4:main: STATE_DIR=/opt/fastorder/bash/scripts/env_app_setup/state
+++ 01-setup-directories.sh:4:main: VERSION_FETCHER_LIB=/opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/lib/version_fetcher.sh
+++ 01-setup-directories.sh:4:main: [[ -f /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/lib/version_fetcher.sh ]]
+++ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/lib/version_fetcher.sh
++ 01-setup-directories.sh:4:main: [[ /opt/fastorder/bash/scripts/env_app_setup/lib/provisioning-init.sh == \s\t\e\p\s\/\0\1\-\s\e\t\u\p\-\d\i\r\e\c\t\o\r\i\e\s\.\s\h ]]
++ 01-setup-directories.sh:4:main: set +e
++ 01-setup-directories.sh:4:main: set +u
++ 01-setup-directories.sh:4:main: set +o pipefail
++ 01-setup-directories.sh:4:main: set +E
+ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh
++ 01-setup-directories.sh:4:main: [[ /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh == \s\t\e\p\s\/\0\1\-\s\e\t\u\p\-\d\i\r\e\c\t\o\r\i\e\s\.\s\h ]]
+ 01-setup-directories.sh:4:main: init_environment
+ 01-setup-directories.sh:4:main: require_bin jq
+ 01-setup-directories.sh:4:main: for b in "$@"
+ 01-setup-directories.sh:4:main: command -v jq
+ 01-setup-directories.sh:4:main: local app_type=general
+ 01-setup-directories.sh:4:main: ENV_ID=zone-universe-main-dev
+ 01-setup-directories.sh:4:main: [[ -z zone-universe-main-dev ]]
+ 01-setup-directories.sh:4:main: [[ -z zone-universe-main-dev ]]
+ 01-setup-directories.sh:4:main: ENV_ID=zone-universe-main-dev
+ 01-setup-directories.sh:4:main: [[ -z zone-universe-main-dev ]]
+ 01-setup-directories.sh:4:main: [[ -z zone-universe-main-dev ]]
+ 01-setup-directories.sh:4:main: [[ -z zone-universe-main-dev ]]
++ 01-setup-directories.sh:4:main: env_dir_for zone-universe-main-dev
++ 01-setup-directories.sh:4:main: echo /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev
+ 01-setup-directories.sh:4:main: ENV_DIR=/opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev
++ 01-setup-directories.sh:4:main: topo_path_for zone-universe-main-dev
++ 01-setup-directories.sh:4:main: echo /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: TOPOLOGY_JSON=/opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: [[ ! -f /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json ]]
+ 01-setup-directories.sh:4:main: validate_topology_json /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: local topo=/opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: [[ -r /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json ]]
+ 01-setup-directories.sh:4:main: jq -e '
    .schema_version == 1
    and (.general.id        | type=="string")
    and (.general.shared_ip | type=="string")
    and (.general.service   | type=="string")
    and (.general.zone    | type=="string")
    and (.general.env       | type=="string")
  ' /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
++ 01-setup-directories.sh:4:main: jq -r .general.service /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: SERVICE=zone
++ 01-setup-directories.sh:4:main: jq -r .general.zone /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: ZONE=universe
++ 01-setup-directories.sh:4:main: jq -r .general.branch /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: BRANCH=main
++ 01-setup-directories.sh:4:main: jq -r .general.env /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: ENV=dev
++ 01-setup-directories.sh:4:main: jq -r '.general.es_nodes_num // 3' /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: ES_NODES_NUM=1
++ 01-setup-directories.sh:4:main: jq -r '.general.pg_workers_num // 3' /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: PG_WORKERS_NUM=1
++ 01-setup-directories.sh:4:main: jq -r '.general.pg_WORKERS_STANDBY_NUM // 3' /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: PG_WORKERS_STANDBY_NUM=3
++ 01-setup-directories.sh:4:main: jq -r '.general.pg_citus_enabled // "yes"' /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: PG_CITUS_ENABLED=yes
+ 01-setup-directories.sh:4:main: [[ general != \g\e\n\e\r\a\l ]]
++ 01-setup-directories.sh:4:main: jq -r .general.shared_ip /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: VM_IP=142.93.238.16
+ 01-setup-directories.sh:4:main: [[ general != \g\e\n\e\r\a\l ]]
++ 01-setup-directories.sh:4:main: jq -r '.general.shared_iface // empty' /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: IFACE=eth0:16
+ 01-setup-directories.sh:4:main: local FINAL_VM_IP=142.93.238.16
+ 01-setup-directories.sh:4:main: set -a
+ 01-setup-directories.sh:4:main: [[ -r /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/generated/general.env ]]
+ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/generated/general.env
++ 01-setup-directories.sh:4:main: ENV_ID=zone-universe-main-dev
++ 01-setup-directories.sh:4:main: SERVICE=zone
++ 01-setup-directories.sh:4:main: zone=universe
++ 01-setup-directories.sh:4:main: BRANCH=main
++ 01-setup-directories.sh:4:main: ENV=dev
++ 01-setup-directories.sh:4:main: VM_IP=142.93.238.16
++ 01-setup-directories.sh:4:main: IFACE=eth0:16
++ 01-setup-directories.sh:4:main: ROOT_DIR=/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
++ 01-setup-directories.sh:4:main: ENV_DIR=/opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev
++ 01-setup-directories.sh:4:main: TOPOLOGY_JSON=/opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
++ 01-setup-directories.sh:4:main: LOG_LEVEL=info
++ 01-setup-directories.sh:4:main: DEBUG_MODE=false
+ 01-setup-directories.sh:4:main: set +a
+ 01-setup-directories.sh:4:main: VM_IP=142.93.238.16
+ 01-setup-directories.sh:4:main: export ENV_ID SERVICE ZONE BRANCH ENV VM_IP IFACE ENV_DIR TOPOLOGY_JSON
+ 01-setup-directories.sh:4:main: export ES_NODES_NUM PG_WORKERS_NUM PG_WORKERS_STANDBY_NUM PG_CITUS_ENABLED
+ 01-setup-directories.sh:4:main: [[ general != \g\e\n\e\r\a\l ]]
+ 01-setup-directories.sh:4:main: info 'Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)'
+ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)'
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
+ 01-setup-directories.sh:4:main: return 0
+ 01-setup-directories.sh:4:main: SERVICE=zone
+ 01-setup-directories.sh:4:main: ZONE=universe
+ 01-setup-directories.sh:4:main: BRANCH=main
+ 01-setup-directories.sh:4:main: ENV=dev
++ 01-setup-directories.sh:4:main: env_id
++ 01-setup-directories.sh:4:main: '[' zone = auth ']'
++ 01-setup-directories.sh:4:main: '[' zone = item ']'
++ 01-setup-directories.sh:4:main: echo zone-universe-main-dev
+ 01-setup-directories.sh:4:main: ENV_ID=zone-universe-main-dev
+ 01-setup-directories.sh:4:main: env=zone-universe-main-dev
+ 01-setup-directories.sh:4:main: nodes=1
+ 01-setup-directories.sh:4:main: [[ 1 =~ ^[1-9][0-9]*$ ]]
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /etc/elasticsearch
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /data/elasticsearch
[2026-02-05 09:42:14 UTC] USER=www-data EUID=0 PID=585374 ACTION=fsop ARGS=mkdir -p /data/elasticsearch
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /var/log/elasticsearch
[2026-02-05 09:42:15 UTC] USER=www-data EUID=0 PID=585383 ACTION=fsop ARGS=mkdir -p /var/log/elasticsearch
+ 01-setup-directories.sh:4:main: APP_NAME=search
+ 01-setup-directories.sh:4:main: TOPOLOGY_FILE=/opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: command -v jq
+ 01-setup-directories.sh:4:main: [[ -f /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json ]]
++ 01-setup-directories.sh:4:main: jq -r --arg app search '.applications[$app].vm_ip // empty' /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: COORD_IP=10.100.1.249
+ 01-setup-directories.sh:4:main: [[ -z 10.100.1.249 ]]
+ 01-setup-directories.sh:4:main: [[ 10.100.1.249 == \n\u\l\l ]]
++ 01-setup-directories.sh:4:main: get_application_domain search
++ 01-setup-directories.sh:4:main: local app_type=search
++ 01-setup-directories.sh:4:main: [[ search == \g\e\n\e\r\a\l ]]
++ 01-setup-directories.sh:4:main: jq -r --arg app search '.applications[$app].domain // empty' /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: COORD_DOMAIN=search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com
+ 01-setup-directories.sh:4:main: info 'Coordinator exists: search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.249)'
+ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Coordinator exists: search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.249)'
[INFO] Coordinator exists: search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.249)
+ 01-setup-directories.sh:4:main: (( i=1 ))
+ 01-setup-directories.sh:4:main: (( i<=nodes ))
++ 01-setup-directories.sh:4:main: printf %02d 1
+ 01-setup-directories.sh:4:main: node_num=01
+ 01-setup-directories.sh:4:main: IDENTIFIER=node-01
+ 01-setup-directories.sh:4:main: APP_NAME=search-node-01
+ 01-setup-directories.sh:4:main: read -r NODE_IP NODE_DOMAIN
++ 01-setup-directories.sh:4:main: setup_directories_per_node node-01 search-node-01
++ 01-setup-directories.sh:4:main: local IDENTIFIER=node-01
++ 01-setup-directories.sh:4:main: local APP_NAME=search-node-01
++ 01-setup-directories.sh:4:main: local env
+++ 01-setup-directories.sh:4:main: env_id
+++ 01-setup-directories.sh:4:main: '[' zone = auth ']'
+++ 01-setup-directories.sh:4:main: '[' zone = item ']'
+++ 01-setup-directories.sh:4:main: echo zone-universe-main-dev
++ 01-setup-directories.sh:4:main: env=zone-universe-main-dev
++ 01-setup-directories.sh:4:main: local TOPOLOGY_FILE=/opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
++ 01-setup-directories.sh:4:main: info 'Setting up Elasticsearch node: node-01'
++ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Setting up Elasticsearch node: node-01'
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /etc/elasticsearch/zone-universe-main-dev/node-01 /etc/elasticsearch/zone-universe-main-dev-node-01
++ 01-setup-directories.sh:4:main: local NODE_IP NODE_DOMAIN
+++ 01-setup-directories.sh:4:main: jq -r --arg app search-node-01 '.applications[$app].vm_ip // empty' /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
[2026-02-05 09:42:15 UTC] USER=www-data EUID=0 PID=585403 ACTION=fsop ARGS=ln -sfn /etc/elasticsearch/zone-universe-main-dev/node-01 /etc/elasticsearch/zone-universe-main-dev-node-01
+ 01-setup-directories.sh:4:main: [[ 1 -eq 1 ]]
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /etc/elasticsearch/zone-universe-main-dev/node-01 /etc/elasticsearch/current
++ 01-setup-directories.sh:4:main: NODE_IP=10.100.1.250
++ 01-setup-directories.sh:4:main: [[ -z 10.100.1.250 ]]
++ 01-setup-directories.sh:4:main: [[ 10.100.1.250 == \n\u\l\l ]]
+++ 01-setup-directories.sh:4:main: get_application_domain search-node-01
+++ 01-setup-directories.sh:4:main: local app_type=search-node-01
+++ 01-setup-directories.sh:4:main: [[ search-node-01 == \g\e\n\e\r\a\l ]]
+++ 01-setup-directories.sh:4:main: jq -r --arg app search-node-01 '.applications[$app].domain // empty' /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
[2026-02-05 09:42:15 UTC] USER=www-data EUID=0 PID=585412 ACTION=fsop ARGS=ln -sfn /etc/elasticsearch/zone-universe-main-dev/node-01 /etc/elasticsearch/current
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /data/elasticsearch/zone-universe-main-dev/node-01 /data/elasticsearch/current
++ 01-setup-directories.sh:4:main: NODE_DOMAIN=search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com
++ 01-setup-directories.sh:4:main: info 'Using existing node-01: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.250)'
++ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Using existing node-01: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.250)'
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh: line 13: printf: write error: Broken pipe
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /etc/elasticsearch/zone-universe-main-dev/node-01/certs
[2026-02-05 09:42:15 UTC] USER=www-data EUID=0 PID=585423 ACTION=fsop ARGS=ln -sfn /data/elasticsearch/zone-universe-main-dev/node-01 /data/elasticsearch/current
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /var/log/elasticsearch/zone-universe-main-dev/node-01 /var/log/elasticsearch/current
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /data/elasticsearch/zone-universe-main-dev/node-01/tmp
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /var/log/elasticsearch/zone-universe-main-dev/node-01
[2026-02-05 09:42:15 UTC] USER=www-data EUID=0 PID=585450 ACTION=fsop ARGS=ln -sfn /var/log/elasticsearch/zone-universe-main-dev/node-01 /var/log/elasticsearch/current
+ 01-setup-directories.sh:4:main: (( i++ ))
+ 01-setup-directories.sh:4:main: (( i<=nodes ))
+ 01-setup-directories.sh:4:main: success 'Directory structure created for '\''zone-universe-main-dev'\'' with 1 node(s).'
+ 01-setup-directories.sh:4:main: printf '[ OK ] %s\n' 'Directory structure created for '\''zone-universe-main-dev'\'' with 1 node(s).'
[ OK ] Directory structure created for 'zone-universe-main-dev' with 1 node(s).
Executing: steps/02-install-dependencies.sh
=== Step 2: Installing/Validating Elasticsearch (latest) ===
++ 01-setup-directories.sh:4:main: id -u elasticsearch
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chown -R elasticsearch:elasticsearch /etc/elasticsearch/zone-universe-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chown -R elasticsearch:elasticsearch /data/elasticsearch/zone-universe-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chown -R elasticsearch:elasticsearch /var/log/elasticsearch/zone-universe-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chmod 0750 /etc/elasticsearch/zone-universe-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chmod 0750 /data/elasticsearch/zone-universe-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chmod 0750 /var/log/elasticsearch/zone-universe-main-dev/node-01
++ 01-setup-directories.sh:4:main: info 'Created dirs for zone-universe-main-dev/node-01 @ 10.100.1.250'
++ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Created dirs for zone-universe-main-dev/node-01 @ 10.100.1.250'
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh: line 13: printf: write error: Broken pipe
++ 01-setup-directories.sh:4:main: printf '%s\n' 10.100.1.250
/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh: line 59: printf: write error: Broken pipe
++ 01-setup-directories.sh:4:main: printf '%s\n' search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com
/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh: line 60: printf: write error: Broken pipe
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] Cleaning dpkg/apt locks...
[2026-02-05 09:42:15 UTC] USER=www-data EUID=0 PID=585640 ACTION=cleanup-dpkg-locks ARGS=
steps/02-install-dependencies.sh: line 16: 585638 Killed                  command sudo -n "$WRAP" cleanup-dpkg-locks
[2026-02-05 09:42:16 UTC] USER=www-data EUID=0 PID=585650 ACTION=fsop ARGS=mkdir -p /etc/apt/keyrings
[2026-02-05 09:42:16 UTC] USER=www-data EUID=0 PID=585660 ACTION=fsop ARGS=chmod 0755 /etc/apt/keyrings
[INFO] apt-get update…
[2026-02-05 09:42:16 UTC] USER=www-data EUID=0 PID=585670 ACTION=pkg ARGS=update
Hit:1 http://apt.postgresql.org/pub/repos/apt jammy-pgdg InRelease
Hit:2 https://packages.confluent.io/deb/7.6 stable InRelease
Hit:3 https://artifacts.elastic.co/packages/8.x/apt stable InRelease
Hit:4 https://deb.nodesource.com/node_22.x nodistro InRelease
Hit:5 https://ppa.launchpadcontent.net/ondrej/php/ubuntu jammy InRelease
Hit:6 https://packages.microsoft.com/repos/azure-cli jammy InRelease
Hit:7 https://apt.grafana.com stable InRelease
Hit:8 https://mirrors.edge.kernel.org/ubuntu jammy InRelease
Hit:9 https://mirrors.edge.kernel.org/ubuntu jammy-backports InRelease
Get:10 https://mirrors.edge.kernel.org/ubuntu jammy-security InRelease [129 kB]
Get:11 https://mirrors.edge.kernel.org/ubuntu jammy-updates InRelease [128 kB]
Hit:12 https://packages.clickhouse.com/deb stable InRelease
Hit:13 https://repos.citusdata.com/community/ubuntu jammy InRelease
Get:14 https://mirrors.edge.kernel.org/ubuntu jammy-updates/main amd64 Packages [3,196 kB]
Get:15 https://mirrors.edge.kernel.org/ubuntu jammy-updates/universe amd64 Packages [1,252 kB]
Fetched 4,705 kB in 4s (1,153 kB/s)
Reading package lists...
[INFO] Installed version : 8.19.10
[INFO] Candidate version : 8.19.11
[INFO] Newer candidate available → upgrading to 8.19.11…
[2026-02-05 09:42:38 UTC] USER=www-data EUID=0 PID=587392 ACTION=cleanup-dpkg-locks ARGS=
steps/02-install-dependencies.sh: line 16: 587390 Killed                  command sudo -n "$WRAP" cleanup-dpkg-locks
[2026-02-05 09:42:38 UTC] USER=www-data EUID=0 PID=587402 ACTION=pkg ARGS=install elasticsearch
Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  php-igbinary php8.4-igbinary php8.4-redis
Use 'sudo apt autoremove' to remove them.
The following packages will be upgraded:
  elasticsearch
1 upgraded, 0 newly installed, 0 to remove and 156 not upgraded.
Need to get 674 MB of archives.
After this operation, 5,357 kB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 elasticsearch amd64 8.19.11 [674 MB]
Fetched 674 MB in 26s (26.4 MB/s)
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 388864 files and directories currently installed.)
Preparing to unpack .../elasticsearch_8.19.11_amd64.deb ...
Unpacking elasticsearch (8.19.11) over (8.19.10) ...
Setting up elasticsearch (8.19.11) ...
Job for systemd-sysctl.service failed because the control process exited with error code.
See "systemctl status systemd-sysctl.service" and "journalctl -xeu systemd-sysctl.service" for details.
packages have been installed but needrestart is suspended
✅ Elasticsearch installation validated.
🎉 Dependencies installed and up-to-date.
Executing: steps/03-create-env-configs.sh
=== Step 3: Creating environment configurations (master + nodes, TLS, units) ===
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Configuring env: zone-universe-main-dev (nodes: 1, http: 9200, transport: 9300)
Using heap size: 1024m per node
[2026-02-05 09:43:41 UTC] USER=www-data EUID=0 PID=590046 ACTION=fsop ARGS=chown root:root /etc/default/elasticsearch
[2026-02-05 09:43:41 UTC] USER=www-data EUID=0 PID=590055 ACTION=fsop ARGS=chmod 0644 /etc/default/elasticsearch
[2026-02-05 09:43:41 UTC] USER=www-data EUID=0 PID=590076 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/jvm.options
[2026-02-05 09:43:41 UTC] USER=www-data EUID=0 PID=590085 ACTION=fsop ARGS=chmod 0644 /etc/elasticsearch/jvm.options
[2026-02-05 09:43:41 UTC] USER=www-data EUID=0 PID=590103 ACTION=fsop ARGS=mkdir -p /etc/systemd/system.conf.d /etc/systemd/user.conf.d
[2026-02-05 09:43:41 UTC] USER=www-data EUID=0 PID=590131 ACTION=passthru ARGS=systemctl daemon-reload
Current max_map_count: 262144
Current swappiness:   1
[2026-02-05 09:43:42 UTC] USER=www-data EUID=0 PID=590218 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/log4j2.properties
[2026-02-05 09:43:42 UTC] USER=www-data EUID=0 PID=590228 ACTION=fsop ARGS=chmod 0644 /etc/elasticsearch/log4j2.properties
[2026-02-05 09:43:42 UTC] USER=www-data EUID=0 PID=590237 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/zone-universe-main-dev/template
[2026-02-05 09:43:42 UTC] USER=www-data EUID=0 PID=590247 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/zone-universe-main-dev /etc/elasticsearch/zone-universe-main-dev/template
[2026-02-05 09:43:42 UTC] USER=www-data EUID=0 PID=590256 ACTION=fsop ARGS=chmod 0755 /etc/elasticsearch/zone-universe-main-dev
[2026-02-05 09:43:42 UTC] USER=www-data EUID=0 PID=590265 ACTION=fsop ARGS=cp /etc/elasticsearch/jvm.options /etc/elasticsearch/zone-universe-main-dev/template/jvm.options
[INFO] 🌐 Registering general environment domain: zone-universe-main-dev.fastorder.com
[INFO]   Allocated VM IP: 10.100.1.55 for general environment
[INFO]   Configuring VM IP 10.100.1.55 on network interface...
[WARNING]   VM IP may already be configured or need manual setup
[WARNING]   Warning: VM IP 10.100.1.55 not found on network interfaces
[ OK ] ✅ Registered general domain zone-universe-main-dev.fastorder.com -> 10.100.1.55
[ OK ] ✅ DNS resolution verified for zone-universe-main-dev.fastorder.com
[INFO] → Configuring zone-universe-main-dev-node-01 (10.100.1.250) roles=[ master, data, data_hot, data_content, ingest ]
[2026-02-05 09:43:43 UTC] USER=www-data EUID=0 PID=590408 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/zone-universe-main-dev/node-01/certs /data/elasticsearch/zone-universe-main-dev/node-01/tmp /var/log/elasticsearch/zone-universe-main-dev/node-01
[2026-02-05 09:43:43 UTC] USER=www-data EUID=0 PID=590417 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/zone-universe-main-dev/node-01
[2026-02-05 09:43:43 UTC] USER=www-data EUID=0 PID=590426 ACTION=fsop ARGS=chmod 0750 /etc/elasticsearch/zone-universe-main-dev/node-01 /data/elasticsearch/zone-universe-main-dev/node-01 /var/log/elasticsearch/zone-universe-main-dev/node-01
[2026-02-05 09:43:44 UTC] USER=www-data EUID=0 PID=590435 ACTION=fsop ARGS=cp /etc/elasticsearch/zone-universe-main-dev/template/jvm.options /etc/elasticsearch/zone-universe-main-dev/node-01/jvm.options
[2026-02-05 09:43:44 UTC] USER=www-data EUID=0 PID=590444 ACTION=fsop ARGS=sed -i s/^-Xms.*/-Xms1024m/ /etc/elasticsearch/zone-universe-main-dev/node-01/jvm.options
[2026-02-05 09:43:44 UTC] USER=www-data EUID=0 PID=590453 ACTION=fsop ARGS=sed -i s/^-Xmx.*/-Xmx1024m/ /etc/elasticsearch/zone-universe-main-dev/node-01/jvm.options
[2026-02-05 09:43:44 UTC] USER=www-data EUID=0 PID=590472 ACTION=fsop ARGS=cp /etc/elasticsearch/log4j2.properties /etc/elasticsearch/zone-universe-main-dev/node-01/log4j2.properties
[2026-02-05 09:43:44 UTC] USER=www-data EUID=0 PID=590509 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/zone-universe-main-dev/node-01
[2026-02-05 09:43:44 UTC] USER=www-data EUID=0 PID=590518 ACTION=fsop ARGS=chmod 0644 /etc/elasticsearch/zone-universe-main-dev/node-01/elasticsearch.yml
[2026-02-05 09:43:44 UTC] USER=www-data EUID=0 PID=590536 ACTION=fsop ARGS=chmod 0644 /etc/default/elasticsearch-zone-universe-main-dev-node-01
[2026-02-05 09:43:44 UTC] USER=www-data EUID=0 PID=590560 ACTION=passthru ARGS=ip addr add 10.100.1.250/32 dev eth0 label eth0:250
[2026-02-05 09:43:44 UTC] USER=www-data EUID=0 PID=590569 ACTION=fsop ARGS=sed -i /[[:space:]]search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com\([[:space:]]\|$\)/d /etc/hosts
[INFO]   → Also added short domain: search-zone-universe-main-dev.fastorder.com
[INFO] ✔ Created configuration for zone-universe-main-dev/node-01 (roles=single-node)
[2026-02-05 09:43:44 UTC] USER=www-data EUID=0 PID=590599 ACTION=fsop ARGS=sed -i /[[:space:]]search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com\([[:space:]]\|$\)/d /etc/hosts
[INFO] ✔ Registered master domain search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com -> 10.100.1.250 (points to node-01)
[INFO] Cleaning up legacy non-templated elasticsearch-*.service units (if any)...
[INFO] No legacy units found.
[INFO] Base template exists: elasticsearch@.service
[ OK ] Created unit: elasticsearch@zone-universe-main-dev-node-01.service
[2026-02-05 09:43:44 UTC] USER=www-data EUID=0 PID=590651 ACTION=passthru ARGS=systemctl daemon-reload

[ OK ] Environment configurations (master + nodes with TLS) created successfully!
[INFO] Environment: zone-universe-main-dev
[INFO] Nodes: 1
[INFO] HTTP Port: 9200
[INFO] Transport Port: 9300
[INFO] Heap Size: 1024m per node
[INFO] Master: search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.249)
[INFO]   node-01: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.250)
[INFO] Systemd units prepared (not started). Start sequence runs in Step 7.
Executing: steps/04-start-clusters.sh
=== Step 7: Starting Elasticsearch clusters (with waits) ===
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Starting Elasticsearch cluster for environment: zone-universe-main-dev (1 nodes)
[INFO] === Ensuring VM IP services are started ===
[WARNING] VM IP service vm-ip-10-100-1-250.service not found - IP might not persist
[INFO] Manually configuring IP: 10.100.1.250
[2026-02-05 09:43:48 UTC] USER=www-data EUID=0 PID=590807 ACTION=configure-network-interface ARGS=lo:search01 10.100.1.250
[INFO] Cleaning up any existing Elasticsearch processes and lock files...
[2026-02-05 09:43:49 UTC] USER=www-data EUID=0 PID=590877 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@zone-universe-main-dev-node-01.service
[INFO] Stopping Elasticsearch services for environment: zone-universe-main-dev ...
[INFO] No active Elasticsearch services found for environment: zone-universe-main-dev
[INFO] Removing lock files from: /data/elasticsearch/zone-universe-main-dev/node-01
[2026-02-05 09:43:49 UTC] USER=www-data EUID=0 PID=590897 ACTION=fsop ARGS=find /data/elasticsearch/zone-universe-main-dev/node-01 -name *.lock -delete
[2026-02-05 09:43:49 UTC] USER=www-data EUID=0 PID=590908 ACTION=fsop ARGS=find /data/elasticsearch/zone-universe-main-dev/node-01 -name node.lock -delete
[2026-02-05 09:43:49 UTC] USER=www-data EUID=0 PID=590921 ACTION=fsop ARGS=find /data/elasticsearch/zone-universe-main-dev/node-01 -name _state -type d -exec rm -rf {} +
[2026-02-05 09:43:49 UTC] USER=www-data EUID=0 PID=590931 ACTION=fsop ARGS=find /tmp -name *elasticsearch*zone-universe-main-dev-node-01* -delete
[ OK ] Cleanup completed for environment: zone-universe-main-dev
[INFO] Checking for port conflicts before starting Elasticsearch...
[INFO] Checking for port conflicts on 10.100.1.249:9200 and 10.100.1.249:9300...
[ OK ] ✓ Ports 9200 and 9300 are available on 10.100.1.249
[INFO] Ensuring correct ownership of Elasticsearch directories...
[2026-02-05 09:43:51 UTC] USER=www-data EUID=0 PID=591157 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch
[2026-02-05 09:43:52 UTC] USER=www-data EUID=0 PID=591168 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /data/elasticsearch
[2026-02-05 09:43:52 UTC] USER=www-data EUID=0 PID=591181 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /var/log/elasticsearch
[ OK ] Directory ownership fixed
[INFO] === Starting Elasticsearch Nodes ===
[INFO] Starting 1 node(s) for cluster
▶ Starting elasticsearch@zone-universe-main-dev-node-01.service (search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200)
[2026-02-05 09:43:52 UTC] USER=www-data EUID=0 PID=591217 ACTION=passthru ARGS=systemctl is-enabled --quiet elasticsearch@zone-universe-main-dev-node-01.service
[2026-02-05 09:43:53 UTC] USER=www-data EUID=0 PID=591280 ACTION=passthru ARGS=systemctl start elasticsearch@zone-universe-main-dev-node-01.service
⏳ Waiting for TCP search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200 to be accessible (timeout 360s)...
✅ Port 9200 is accessible on search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com.
⏳ Waiting for ES HTTP readiness on http://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200 (timeout 300s)...
[ OK ] ES HTTP ready on search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
[ OK ] elasticsearch@zone-universe-main-dev-node-01.service is up and answering HTTP on search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Node 1 started successfully
[INFO] Cluster with 1 node(s) started successfully
⏳ Waiting for the cluster to elect master and settle...
⏳ Waiting for cluster health=green via search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200 (timeout 300s)...
[ OK ] Cluster is GREEN (nodes="number_of_nodes") on search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
[ OK ] Cluster zone-universe-main-dev is healthy and green!

[INFO] === Final Status Check ===
[2026-02-05 09:44:46 UTC] USER=www-data EUID=0 PID=593554 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@zone-universe-main-dev-node-01.service
[ OK ] elasticsearch@zone-universe-main-dev-node-01.service is ACTIVE (search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200)
  └── HTTP responding on search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200 ✓
[ OK ] All 1 node(s) in environment 'zone-universe-main-dev' are running successfully!
[INFO] Node endpoints:
  - http://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200


[ OK ] Elasticsearch cluster started successfully!
[INFO] Environment: zone-universe-main-dev
[INFO] Nodes: 1
[INFO] Cluster endpoints:
  - http://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === Quick Cluster Information ===
Cluster Name: fastorder-zone-universe-main-dev
Node Name: zone-universe-main-dev-node-01
Version: 8.19.11
Architecture: 1 node(s), each on default port 9200

Cluster with 1 node(s) started successfully (each on port 9200)
Executing: steps/05-verify-setup.sh
=== Step 8: Verifying setup (with retries) ===
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Verifying environment: zone-universe-main-dev (1 nodes, Single-node)
Main HTTP endpoint: http://search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
Testing network connectivity to search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200...
✓ Domain connection available
Testing HTTP response...
[ OK ] ✓ zone-universe-main-dev is responding on search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200

[INFO] === Cluster Health ===
{
  "cluster_name" : "fastorder-zone-universe-main-dev",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
[ OK ] Cluster status: GREEN ("number_of_nodes" nodes)

[INFO] === Cluster Nodes ===
ip           heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.100.1.250           53          98  96    8.17    7.34     6.99 dhims     *      zone-universe-main-dev-node-01

[INFO] === Single-Node Service Verification ===
Testing coordinator service (search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200)...
  ✓ Coordinator HTTP responding on search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
    Name: zone-universe-main-dev-node-01, Version: 8.19.11

[INFO] === Cluster State Summary ===
Using jq for formatted output:
jq parsing failed

[ OK ] === Verification Summary ===
[INFO] Environment: zone-universe-main-dev
[INFO] Nodes configured: 1
[INFO] Main endpoint: http://search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
[INFO] Service endpoint: http://search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200

[INFO] === Final Connectivity Test ===
  ✓ Coordinator: search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200

[ OK ] Single-node cluster is responding successfully!
[ OK ] Elasticsearch cluster 'zone-universe-main-dev' verification completed successfully!
Executing: steps/06-confirm-working.sh
=== Step 9: Comprehensive Cluster Verification (gated) ===
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
========================================
🔍 Verifying Environment: zone-universe-main-dev (1 nodes)
========================================
Domain: search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com
Environment: zone-universe-main-dev
Nodes: 1

[INFO] Testing network connectivity...
Setup type: Single-node
Testing endpoint: search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
[ OK ] ✓ Using domain: search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com


📡 Coordinator Service (elasticsearch@zone-universe-main-dev-node-01.service)
Endpoint: search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
--------------------------------
[2026-02-05 09:44:48 UTC] USER=www-data EUID=0 PID=593732 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@zone-universe-main-dev-node-01.service
✅ Service: ACTIVE
⏳ Waiting for TCP search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200 to be accessible (timeout 5s)...
✅ Port 9200 is accessible on search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com.
✅ Port: LISTENING on search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
✅ HTTP: RESPONDING on search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
   Node name: zone-universe-main-dev-node-01

========================================
🏥 Cluster Health Check
========================================
Cluster Name: fastorder-zone-universe-main-dev
Nodes Count: "number_of_nodes"
Status: green
[ OK ] ✅ Cluster status: GREEN (healthy)

Full cluster health:
{
  "cluster_name" : "fastorder-zone-universe-main-dev",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

========================================
📊 Final Verification Results
========================================
[ OK ] ✅ Comprehensive verification PASSED!
[ OK ] Environment 'zone-universe-main-dev' with 1 nodes is fully operational

📋 QUICK DIAGNOSTIC COMMANDS:
----------------------------------------
# Test cluster endpoints:
curl http://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

# Check cluster health:
curl http://search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200/_cluster/health?pretty

# Check nodes info:
curl http://search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200/_cat/nodes?v

# Check all Elasticsearch ports:
sudo ss -tlnp | grep java

# Check systemd service status:
sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status elasticsearch@zone-universe-main-dev-node-01.service

# View recent logs:
sudo journalctl -u elasticsearch@zone-universe-main-dev-node-01.service -f

[INFO] Environment: zone-universe-main-dev
[INFO] Nodes: 1
[INFO] Port: 9200 (default Elasticsearch port)
[INFO] Coordinator endpoint: http://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

=== Elasticsearch HTTP Setup completed successfully! ===
Environment:  (1 nodes)
Port: 9200 (default Elasticsearch port)

✅ Coordinator endpoint: http://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

Quick test commands:
  curl http://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
  curl http://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty


✓ Step 1 completed successfully!

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 2: Executing Make Https
Folder: 02-make-https
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

=== Elasticsearch HTTPS Setup ===
Configure HTTPS/SSL for Elasticsearch cluster
[INFO] Using web-provided environment: zone-universe-main-dev
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: 
Nodes: 1
Node: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.250)
Port: 9200 (default port)

Executing: steps/01-generate-ssl-certificates.sh
==================================================================
STEP 1: Generate SSL certificates for Elasticsearch transport
==================================================================
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Domain: zone-universe-main-dev.fastorder.com
Environment: zone-universe-main-dev
Nodes: 1
Per-node VM IPs and domains:
  Node 1: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.250)
Port: 9200 (default port for all nodes)

=== Generating SSL certificates for ES transport ===
[INFO] Generating certificates for environment: zone-universe-main-dev (1 nodes)
[INFO] Configuring certificates for 1 node(s)
[INFO] Certificate storage: /etc/fastorder/elasticsearch/certs/zone-universe-main-dev
[2026-02-05 09:44:49 UTC] USER=www-data EUID=0 PID=593833 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/temp-593805
[2026-02-05 09:44:49 UTC] USER=www-data EUID=0 PID=593842 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/temp-593805
[2026-02-05 09:44:49 UTC] USER=www-data EUID=0 PID=593851 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/temp-593805
[2026-02-05 09:44:49 UTC] USER=www-data EUID=0 PID=593870 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/temp-593805/instances.yml
[INFO] Creating certificate instances configuration...
  Adding node: zone-universe-main-dev-node-01 (search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com, 10.100.1.250)

[INFO] Certificate instances configuration:
instances:
  - name: zone-universe-main-dev-node-01
    dns: [ "zone-universe-main-dev-node-01", "localhost", "search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com" ]
    ip:  [ "10.100.1.250", "127.0.0.1" ]

[INFO] Creating Certificate Authority for zone-universe-main-dev...
[2026-02-05 09:44:49 UTC] USER=www-data EUID=0 PID=593901 ACTION=fsop ARGS=mkdir -p /etc/fastorder/elasticsearch/certs/zone-universe-main-dev/node-01/certs
[2026-02-05 09:44:49 UTC] USER=www-data EUID=0 PID=593920 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/fastorder/elasticsearch/certs/zone-universe-main-dev
[2026-02-05 09:44:49 UTC] USER=www-data EUID=0 PID=593938 ACTION=fsop ARGS=chmod -R 755 /etc/fastorder/elasticsearch/certs/zone-universe-main-dev
yes: standard output: Broken pipe
[ OK ] ✓ CA certificate created

[INFO] Creating node certificates for zone-universe-main-dev...
yes: standard output: Broken pipe
[ OK ] ✓ Node certificates created

[INFO] Distributing certificates...
  Configuring certificates for node 1 (zone-universe-main-dev-node-01)...
[2026-02-05 09:44:59 UTC] USER=www-data EUID=0 PID=594403 ACTION=fsop ARGS=cp /etc/fastorder/elasticsearch/certs/zone-universe-main-dev/node-01/certs/nodes/zone-universe-main-dev-node-01.crt /etc/fastorder/elasticsearch/certs/zone-universe-main-dev/node-01/
[2026-02-05 09:44:59 UTC] USER=www-data EUID=0 PID=594413 ACTION=fsop ARGS=cp /etc/fastorder/elasticsearch/certs/zone-universe-main-dev/node-01/certs/nodes/zone-universe-main-dev-node-01.key /etc/fastorder/elasticsearch/certs/zone-universe-main-dev/node-01/
[2026-02-05 09:44:59 UTC] USER=www-data EUID=0 PID=594422 ACTION=fsop ARGS=chmod 644 /etc/fastorder/elasticsearch/certs/zone-universe-main-dev/node-01/zone-universe-main-dev-node-01.crt
[ OK ]   ✓ Certificates copied for zone-universe-main-dev-node-01
[2026-02-05 09:44:59 UTC] USER=www-data EUID=0 PID=594447 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/fastorder/elasticsearch/certs/zone-universe-main-dev/node-01
[2026-02-05 09:44:59 UTC] USER=www-data EUID=0 PID=594473 ACTION=fsop ARGS=find /etc/fastorder/elasticsearch/certs/zone-universe-main-dev/node-01/certs -type f -name *.key -exec chmod 600 {} ;
[2026-02-05 09:45:00 UTC] USER=www-data EUID=0 PID=594488 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/fastorder/elasticsearch/certs/zone-universe-main-dev/node-01/certs
[2026-02-05 09:45:00 UTC] USER=www-data EUID=0 PID=594505 ACTION=fsop ARGS=rm -rf /etc/elasticsearch/temp-593805
[ OK ] ✓ Certificates ready for environment: zone-universe-main-dev

[ OK ] ✓ SSL certificate generation completed successfully!
[INFO] Environment: zone-universe-main-dev
[INFO] Nodes configured: 1
[INFO] Per-node VM IPs and domains (each with default port 9200):
  Node 1: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.250)
[INFO] Certificate directory: /etc/fastorder/elasticsearch/certs/zone-universe-main-dev/node-01/certs

[INFO] === Certificate Summary ===
CA Certificate: /etc/fastorder/elasticsearch/certs/zone-universe-main-dev/node-01/certs/ca/ca.crt
Node Certificates:
  - zone-universe-main-dev-node-01: /etc/fastorder/elasticsearch/certs/zone-universe-main-dev/node-01/

[INFO] === Verification Commands ===
# Verify CA certificate:
openssl x509 -in /etc/fastorder/elasticsearch/certs/zone-universe-main-dev/node-01/certs/ca/ca.crt -text -noout

# Verify node certificates:
openssl x509 -in /etc/fastorder/elasticsearch/certs/zone-universe-main-dev/node-01/zone-universe-main-dev-node-01.crt -text -noout

[INFO] Next: Configure transport SSL in Elasticsearch configuration files
Executing: steps/02-enable-security-transport.sh
==================================================================
STEP 2: Enable security with transport SSL
==================================================================
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Nodes: 1
Node: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.250)

[INFO] === Single-Node Setup ===
[INFO] Enabling security (xpack.security.enabled: true)
[2026-02-05 09:45:00 UTC] USER=www-data EUID=0 PID=594572 ACTION=fsop ARGS=sed -i /^xpack.security.enabled:/d /etc/elasticsearch/zone-universe-main-dev/node-01/elasticsearch.yml
[INFO] Disabling transport SSL (not needed for single-node)
[2026-02-05 09:45:00 UTC] USER=www-data EUID=0 PID=594593 ACTION=fsop ARGS=sed -i /^xpack.security.transport.ssl.enabled:/d /etc/elasticsearch/zone-universe-main-dev/node-01/elasticsearch.yml

[ OK ] ==================================================================
[ OK ] Security and Transport SSL Configuration Complete
[ OK ] ==================================================================
[INFO] Environment: zone-universe-main-dev
[INFO] Nodes: 1
[INFO] Security enabled: true
[INFO] Transport SSL enabled: false (not required for single-node)

[INFO] === Next Step ===
Restart services to apply security configuration (step 04)
Executing: steps/03-http-ssl.sh
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
==================================================================
Direct HTTPS Configuration (native TLS, PEM)
==================================================================
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Nodes: 1
Node: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.250)
Port: 9200 (default port)
Domain: zone-universe-main-dev.fastorder.com

[INFO] === Single-Node Direct HTTPS Setup ===
🔐 Generated PKCS12 password: DnQizP7x... (32 chars)
[INFO] Using first node configuration: /etc/elasticsearch/zone-universe-main-dev/node-01
[2026-02-05 09:45:02 UTC] USER=www-data EUID=0 PID=595162 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/zone-universe-main-dev/node-01
[2026-02-05 09:45:03 UTC] USER=www-data EUID=0 PID=595233 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/zone-universe-main-dev/node-01
[INFO] Checking prerequisites...
[ OK ] ✓ Prerequisites verified
[INFO] Setting up temporary directories...
[2026-02-05 09:45:03 UTC] USER=www-data EUID=0 PID=595290 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs
[2026-02-05 09:45:03 UTC] USER=www-data EUID=0 PID=595365 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs
[2026-02-05 09:45:03 UTC] USER=www-data EUID=0 PID=595426 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/zone-universe-main-dev
[2026-02-05 09:45:03 UTC] USER=www-data EUID=0 PID=595484 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/zone-universe-main-dev/node-01
[2026-02-05 09:45:03 UTC] USER=www-data EUID=0 PID=595631 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out
[ OK ] ✓ Directories created

[INFO] Building certificate instances configuration...
[INFO] Certificate instances configuration:
instances:
  - name: "zone-universe-main-dev-http"
    dns:  [ "localhost", "web-03", "search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com", "zone-universe-main-dev-node-01.fastorder.com", "search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com", "search-zone-universe-main-dev.fastorder.com", "zone-universe-main-dev-node-01.local" ]
    ip:   [ "10.100.1.250", "127.0.0.1", "::1" ]
[ OK ] ✓ Instances configuration created

[INFO] Generating HTTP Certificate Authority...
[2026-02-05 09:45:04 UTC] USER=www-data EUID=0 PID=595770 ACTION=fsop ARGS=rm -f /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/http-ca.zip /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out/http-certs.zip
[2026-02-05 09:45:04 UTC] USER=www-data EUID=0 PID=595856 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs
[2026-02-05 09:45:04 UTC] USER=www-data EUID=0 PID=595915 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs
[2026-02-05 09:45:10 UTC] USER=www-data EUID=0 PID=597619 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/http-ca.zip
Archive:  /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/http-ca.zip
   creating: /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/ca/
  inflating: /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/ca/ca.crt  
  inflating: /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/ca/ca.key  
[ OK ] ✓ HTTP CA generated successfully

[INFO] Generating per-node HTTP certificates...
[2026-02-05 09:45:10 UTC] USER=www-data EUID=0 PID=597653 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out
[2026-02-05 09:45:10 UTC] USER=www-data EUID=0 PID=597704 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out
[2026-02-05 09:45:26 UTC] USER=www-data EUID=0 PID=601065 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out/http-certs.zip
[2026-02-05 09:45:26 UTC] USER=www-data EUID=0 PID=601075 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out/http
Archive:  /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out/http-certs.zip
   creating: /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out/http/zone-universe-main-dev-http/
  inflating: /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out/http/zone-universe-main-dev-http/zone-universe-main-dev-http.crt  
  inflating: /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out/http/zone-universe-main-dev-http/zone-universe-main-dev-http.key  
[ OK ] ✓ HTTP certificates generated successfully

[INFO] Installing certificates and configuring services...
[INFO] Configuring main service for single-node HTTPS...
[2026-02-05 09:45:26 UTC] USER=www-data EUID=0 PID=601092 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/zone-universe-main-dev/node-01/certs
[2026-02-05 09:45:26 UTC] USER=www-data EUID=0 PID=601119 ACTION=fsop ARGS=cp /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out/http/zone-universe-main-dev-http/zone-universe-main-dev-http.crt /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http.crt
[2026-02-05 09:45:26 UTC] USER=www-data EUID=0 PID=601133 ACTION=fsop ARGS=cp /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out/http/zone-universe-main-dev-http/zone-universe-main-dev-http.key /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http.key
[2026-02-05 09:45:26 UTC] USER=www-data EUID=0 PID=601146 ACTION=fsop ARGS=cp /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/ca/ca.crt /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http_ca.crt
[2026-02-05 09:45:26 UTC] USER=www-data EUID=0 PID=601157 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/zone-universe-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-02-05 09:45:26 UTC] USER=www-data EUID=0 PID=601170 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/zone-universe-main-dev/node-01/certs
[2026-02-05 09:45:26 UTC] USER=www-data EUID=0 PID=601183 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http.key
[2026-02-05 09:45:26 UTC] USER=www-data EUID=0 PID=601202 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http.crt /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http_ca.crt /etc/elasticsearch/zone-universe-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-02-05 09:45:27 UTC] USER=www-data EUID=0 PID=601212 ACTION=fsop ARGS=sed -i -E -e /^\s*xpack\.security\.http\.ssl(\..*)?\s*:/d /etc/elasticsearch/zone-universe-main-dev/node-01/elasticsearch.yml
[2026-02-05 09:45:27 UTC] USER=www-data EUID=0 PID=601222 ACTION=fsop ARGS=sed -i /^# --- BEGIN direct HTTPS (managed, PEM) ---$/,/^# --- END direct HTTPS (managed, PEM) ---$/d /etc/elasticsearch/zone-universe-main-dev/node-01/elasticsearch.yml
[2026-02-05 09:45:27 UTC] USER=www-data EUID=0 PID=601231 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/zone-universe-main-dev/node-01/certs/fastorder_ra_root.crt
YAML: /etc/elasticsearch/zone-universe-main-dev/node-01/elasticsearch.yml
[ OK ]   ✓ Main service configured with HTTPS
[2026-02-05 09:45:27 UTC] USER=www-data EUID=0 PID=601250 ACTION=fsop ARGS=rm -rf /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients
[2026-02-05 09:45:27 UTC] USER=www-data EUID=0 PID=601265 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients
[2026-02-05 09:45:27 UTC] USER=www-data EUID=0 PID=601276 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients
Archive:  /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client.zip
   creating: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/
  inflating: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.crt  
  inflating: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.key  
[INFO] Creating P12 keystore for es-client...
[2026-02-05 09:45:31 UTC] USER=www-data EUID=0 PID=601470 ACTION=fsop ARGS=mv /tmp/es-client-594645.p12 /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[ OK ] ✓ Created P12 keystore: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-02-05 09:45:31 UTC] USER=www-data EUID=0 PID=601479 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients
[2026-02-05 09:45:31 UTC] USER=www-data EUID=0 PID=601488 ACTION=fsop ARGS=chmod 640 /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.key
[2026-02-05 09:45:31 UTC] USER=www-data EUID=0 PID=601497 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-02-05 09:45:31 UTC] USER=www-data EUID=0 PID=601506 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.crt /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http_ca.crt
[INFO] Saving keystore passwords to secrets vault...
🔑 Configuring AWS credentials for secrets vault...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[INFO] 🔐 Vaulting search passwords to remote backend...
✅ Passwords vaulted to remote backend
✓ Keystore passwords saved to secrets vault: search/zone-universe-main-dev/keystore-passwords

[INFO] === Installing CA Certificate for Users ===
[INFO] HOME not set, skipping user CA installation

✓ Direct HTTPS configuration completed for environment: zone-universe-main-dev
[INFO] All services now serve HTTPS using PEM certificates
[INFO] Network binding: 10.100.1.250
[INFO] HTTPS endpoint: https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === Certificate Summary ===
CA Certificate: /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/ca/ca.crt
Certificate Directory: /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs
Main service certificates: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/

[INFO] === Next Steps ===
1. Restart Elasticsearch services to apply HTTPS configuration
2. Test HTTPS connectivity: curl https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
3. Verify certificates: openssl s_client -connect search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === SSL Certificate Information ===
• CA certificate installed for user - curl will work without -k flag
• Each certificate includes node-specific domain, VM IP, and localhost in Subject Alternative Names
• Certificates are valid for 3 years from generation date

[WARNING] Important: You'll need to restart Elasticsearch services for HTTPS to take effect
Executing: steps/04-restart-systemd-services.sh
==================================================================
STEP 4 (STRICT): Restart systemd services and verify secure health
==================================================================
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: zone-universe-main-dev
Nodes: 1
Per-node endpoints (all use default port 9200):
  Node 1: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.250)

[INFO] Building service list for environment: zone-universe-main-dev (1 nodes)
  - elasticsearch@zone-universe-main-dev-node-01.service (port 9200)

[INFO] Will restart 1 service(s) for environment: zone-universe-main-dev
[2026-02-05 09:45:35 UTC] USER=www-data EUID=0 PID=601742 ACTION=passthru ARGS=systemctl daemon-reload

[INFO] === Ensuring VM IPs are configured correctly ===
[INFO] ✓ 10.100.1.250 already configured on eth0 for node-01

[INFO] === Ensuring transport SSL certificates for all nodes ===
[INFO] ✓ Transport certificates already exist for node-01

[INFO] === Restarting Services ===
↻ Restarting elasticsearch@zone-universe-main-dev-node-01.service ...
[2026-02-05 09:45:36 UTC] USER=www-data EUID=0 PID=601804 ACTION=passthru ARGS=systemctl restart elasticsearch@zone-universe-main-dev-node-01.service
[2026-02-05 09:45:41 UTC] USER=www-data EUID=0 PID=602134 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@zone-universe-main-dev-node-01.service
[ OK ] elasticsearch@zone-universe-main-dev-node-01.service is active
[INFO] Waiting 10s for Elasticsearch to start listening on ports...

[INFO] === Waiting for STRICT Secure Cluster Health ===
[INFO] Waiting for port 9200 on 10.100.1.250 (timeout 120s)...
[INFO] Waiting for cluster to form and be ready for write operations...
✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
[INFO] Cluster stable and ready for operations (20 consecutive healthy responses over 40s)

[INFO] Performing final cluster health check before password setup...
[INFO] Elastic password not found, running password setup...
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
╔════════════════════════════════════════════════════════════╗
║   Elasticsearch Password Management via AWS Secrets MGR   ║
╚════════════════════════════════════════════════════════════╝

Environment: zone-universe-main-dev
User:        elastic
Identifier:  node-01
AWS Secret:  fastorder/search/zone/universe/main/dev/elasticsearch/node-01

Using configuration path: /etc/elasticsearch/zone-universe-main-dev/node-01 (IDENTIFIER: node-01)
Node domain: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com
HTTP port: 9200 (default Elasticsearch port)
[INFO] xpack.security.enabled already true → no restart.
[INFO] No restart needed.
[2026-02-05 09:47:34 UTC] USER=www-data EUID=0 PID=606765 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/zone-universe-main-dev/node-01
[2026-02-05 09:47:34 UTC] USER=www-data EUID=0 PID=606792 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/zone-universe-main-dev/node-01/users /etc/elasticsearch/zone-universe-main-dev/node-01/users_roles
[2026-02-05 09:47:34 UTC] USER=www-data EUID=0 PID=606801 ACTION=fsop ARGS=chmod 660 /etc/elasticsearch/zone-universe-main-dev/node-01/users /etc/elasticsearch/zone-universe-main-dev/node-01/users_roles
✓ users/users_roles present and writable
[2026-02-05 09:47:34 UTC] USER=www-data EUID=0 PID=606811 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/zone-universe-main-dev/node-01/elasticsearch.keystore
[2026-02-05 09:47:34 UTC] USER=www-data EUID=0 PID=606820 ACTION=fsop ARGS=chmod 660 /etc/elasticsearch/zone-universe-main-dev/node-01/elasticsearch.keystore
✓ Keystore exists: /etc/elasticsearch/zone-universe-main-dev/node-01/elasticsearch.keystore
HTTPS is enabled in configuration
✓ Found HTTP CA certificate: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http_ca.crt
✓ Using client certificates for mTLS
Waiting for Elasticsearch to be reachable at https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200...
✓ Elasticsearch is reachable (HTTP 401)

ES_PATH_CONF: /etc/elasticsearch/zone-universe-main-dev/node-01
HTTP URL:    https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

Running HTTP reset (online, --batch)…
Note: Using HTTPS - tools will read SSL config from elasticsearch.yml
Command output:
Password for the [elastic] user successfully reset.
New value: 1mQB+-YPcyy89quL19Qk
Exit status: 0
✓ HTTP reset succeeded for elastic
Storing credentials in AWS Secrets Manager: fastorder/search/zone/universe/main/dev/elasticsearch/node-01
ℹ️  Setting Elasticsearch credentials in vault: fastorder/search/zone/universe/main/dev/elasticsearch/node-01
ℹ️  Setting secret in AWS Secrets Manager: fastorder/search/zone/universe/main/dev/elasticsearch/node-01
✅ Secret created: fastorder/search/zone/universe/main/dev/elasticsearch/node-01
✅ Elasticsearch credentials set in vault: fastorder/search/zone/universe/main/dev/elasticsearch/node-01
✓ Password stored in AWS Secrets Manager: fastorder/search/zone/universe/main/dev/elasticsearch/node-01
✓ Cache cleared for: fastorder/search/zone/universe/main/dev/elasticsearch/node-01

✓ Done. Password stored in AWS Secrets Manager: fastorder/search/zone/universe/main/dev/elasticsearch/node-01

Usage Examples:
  # Retrieve password using AWS CLI
  aws secretsmanager get-secret-value --secret-id fastorder/search/zone/universe/main/dev/elasticsearch/node-01 --region ${AWS_REGION:-me-central-1}

  # Using fastctl
  fastctl secrets get fastorder/search/zone/universe/main/dev/elasticsearch/node-01

  # Test connection
  curl -u elastic:$(fastctl secrets get fastorder/search/zone/universe/main/dev/elasticsearch/node-01 --field password) https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health
✓ Retrieved password from AWS Secrets Manager
[INFO] Testing cluster at: https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Using SSL CA certificate: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http_ca.crt
[INFO] Using client cert/key for mTLS
[INFO] Using client cert/key for mTLS: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.key
[INFO]   ⏳ waiting for secure cluster health (require 200) at https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200 (timeout 30s)...
[ OK ] Cluster health OK: green
[ OK ] Authentication successful with existing password

==================================================================
[ OK ] All services restarted successfully!
[ OK ] Cluster is healthy, HTTPS-secure, and responding with 200
[INFO] Environment: zone-universe-main-dev
[INFO] Services: 1
[INFO] Endpoint: https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === Manual verification (copy/paste) ===
curl -u 'elastic:1mQB+-YPcyy89quL19Qk' \
  --cacert '/etc/elasticsearch/zone-universe-main-dev/node-01/certs/http_ca.crt' \
  --cert   '/etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.crt' \
  --key    '/etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.key' \
  'https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty'

[INFO] === Quick checks ===
curl -u 'elastic:1mQB+-YPcyy89quL19Qk' --cacert '/etc/elasticsearch/zone-universe-main-dev/node-01/certs/http_ca.crt' --cert '/etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.crt' --key '/etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.key' https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cat/nodes?v
curl -u 'elastic:1mQB+-YPcyy89quL19Qk' --cacert '/etc/elasticsearch/zone-universe-main-dev/node-01/certs/http_ca.crt' --cert '/etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.crt' --key '/etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.key' https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/
Executing: steps/05-test-elastic.sh
==================================================================
STEP 5: Test Elasticsearch Cluster
==================================================================
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: zone-universe-main-dev
Nodes: 1
Node: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.250)
Port: 9200 (default port)

[INFO] Using centralized test suite: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/02-make-https/lib/elasticsearch-test-suite.sh
[INFO] Executing centralized test suite with args: -v -t all --env zone-universe-main-dev -u elastic
[INFO] Using environment from web interface: zone-universe-main-dev
[2026-02-05 09:47:50] Using web-provided environment: zone-universe-main-dev
[2026-02-05 09:47:50] Service: zone, Zone: universe, Branch: main, Env: dev
✓ Environment initialized successfully (mode: general)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
ℹ Using environment from web interface: zone-universe-main-dev
[2026-02-05 09:47:50] Using web-provided environment: zone-universe-main-dev
[2026-02-05 09:47:50] Service: zone, Zone: universe, Branch: main, Env: dev
✓ Environment initialized successfully (mode: general)
ℹ Project root: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch
ℹ Environment:  zone-universe-main-dev
ℹ Nodes count:  1
ℹ Endpoint:      https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
ℹ Using CA:       /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http_ca.crt
ℹ Using mTLS:     /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.key
╔════════════════════════════════════════════╗
║    Elasticsearch Centralized Test Suite    ║
╚════════════════════════════════════════════╝

=== Authentication Test ===
✓ Loaded credentials for user elastic from AWS Secrets Manager
Curl (local): curl --cacert /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http_ca.crt --cert /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.crt --key /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.key  -u 'elastic:********' 'https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty'
✓ Local authentication successful (HTTP 200).
{
  "cluster_name" : "fastorder-zone-universe-main-dev",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 3,
  "active_shards" : 3,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Executing: steps/06-final-testing.sh
==================================================================
STEP 6: Final Testing and Verification
==================================================================
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: zone-universe-main-dev
Nodes: 1
Node: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.250)
Port: 9200 (default port)

[INFO] Using centralized test suite: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/02-make-https/lib/elasticsearch-test-suite.sh
[INFO] Using environment from web interface: zone-universe-main-dev
[2026-02-05 09:47:51] Using web-provided environment: zone-universe-main-dev
[2026-02-05 09:47:51] Service: zone, Zone: universe, Branch: main, Env: dev
✓ Environment initialized successfully (mode: general)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
ℹ Using environment from web interface: zone-universe-main-dev
[2026-02-05 09:47:51] Using web-provided environment: zone-universe-main-dev
[2026-02-05 09:47:51] Service: zone, Zone: universe, Branch: main, Env: dev
✓ Environment initialized successfully (mode: general)
ℹ Project root: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch
ℹ Environment:  zone-universe-main-dev
ℹ Nodes count:  1
ℹ Endpoint:      https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
ℹ Using CA:       /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http_ca.crt
ℹ Using mTLS:     /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.key
╔════════════════════════════════════════════╗
║    Elasticsearch Centralized Test Suite    ║
╚════════════════════════════════════════════╝

=== Authentication Test ===
✓ Loaded credentials for user elastic from AWS Secrets Manager
Curl (local): curl --cacert /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http_ca.crt --cert /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.crt --key /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.key  -u 'elastic:********' 'https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty'
✓ Local authentication successful (HTTP 200).
{
  "cluster_name" : "fastorder-zone-universe-main-dev",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 3,
  "active_shards" : 3,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Executing: steps/07-set-passwords.sh
==================================================================
STEP 7: Setting cluster passwords (bootstrap via alias)
==================================================================
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Using HTTPS with CA: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http_ca.crt (host: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com)
[INFO] Using centralized password setter: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/02-make-https/steps/../lib/elasticsearch-set-password.sh
[ OK ] Elastic password already valid (HTTP 200) via search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com; nothing to do.
Executing: steps/08-create-app-user.sh
==================================================================
STEP 8: Create Application User and Roles (cluster-scoped)
==================================================================
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: zone-universe-main-dev
Nodes: 1

[INFO] Using HTTPS with CA: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http_ca.crt (host: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com)
[ OK ] Retrieved elastic password from Vault (cluster scope).
[INFO] Configuration:
[INFO]   App User         : app_user
[INFO]   Read-only Role   : app_ro
[INFO]   Read-write Role  : app_rw
[INFO]   Index Patterns   : app-*,cdc-*,zone_universe_*,*_account_router
[INFO]   Endpoint         : https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] Creating read-only role: app_ro
[ OK ] ✓ Role app_ro ensured
[INFO] Creating read-write role: app_rw
[ OK ] ✓ Role app_rw ensured
[INFO] Creating/Updating application user: app_user
[ OK ] ✓ User app_user ensured
ℹ️  Setting Elasticsearch credentials in vault: fastorder/search/zone/universe/main/dev/elasticsearch/node-01/app_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/search/zone/universe/main/dev/elasticsearch/node-01/app_user
✅ Secret created: fastorder/search/zone/universe/main/dev/elasticsearch/node-01/app_user
✅ Elasticsearch credentials set in vault: fastorder/search/zone/universe/main/dev/elasticsearch/node-01/app_user
[ OK ] ✓ Stored app_user password under 'node-01/app_user'
ℹ️  Setting Elasticsearch credentials in vault: fastorder/search/zone/universe/main/dev/elasticsearch/cluster/app_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/search/zone/universe/main/dev/elasticsearch/cluster/app_user
✅ Secret created: fastorder/search/zone/universe/main/dev/elasticsearch/cluster/app_user
✅ Elasticsearch credentials set in vault: fastorder/search/zone/universe/main/dev/elasticsearch/cluster/app_user
[ OK ] ✓ Stored app_user password under 'cluster/app_user'
[INFO] Testing authentication for app_user...
[ OK ] ✓ Authentication test passed for app_user

[ OK ] ✓ Application user and roles created successfully!
[INFO] User    : app_user
[INFO] Roles   : app_ro, app_rw
[INFO] Patterns: app-*,cdc-*,zone_universe_*,*_account_router
[INFO] Endpoint: https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
Executing: steps/09-config.sh
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Nodes: 1
Node: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.250)
Port: 9200

✓ Auto mode: Cloud IMDS detected → MODE=role
[INFO] Mode: role

[INFO] AWS Region: me-central-1
[INFO] MODE=role → will purge any static S3 keys from each node keystore

[2026-02-05 09:49:13 UTC] USER=www-data EUID=0 PID=611208 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/zone-universe-main-dev/node-01
[INFO] • node-01 keystore cleared (role-based auth)
[2026-02-05 09:49:25 UTC] USER=www-data EUID=0 PID=611752 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 09:49:26 UTC] USER=www-data EUID=0 PID=611807 ACTION=passthru ARGS=systemctl restart elasticsearch@zone-universe-main-dev-node-01.service
✓ ✓ restarted elasticsearch@zone-universe-main-dev-node-01.service

⏳ Waiting for HTTPS readiness on https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Waiting HTTP readiness at https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/ (200/401/302)…
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
[OK] Ready: 401
⏳ Waiting for cluster health (green|yellow)
[INFO] Waiting health (green|yellow) at https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health…
[OK] 401 pre-auth received; security enabled.
✓ ✓ zone-universe-main-dev is responding via search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com

✓ ✓ AWS S3 configuration completed for environment: zone-universe-main-dev (1 nodes)
[INFO] Mode: role
[INFO] Region: me-central-1
Executing: steps/0ld-03-http-ssl.sh
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
==================================================================
STEP 5: HTTP SSL Configuration (Optional)
==================================================================
Environment: zone-universe-main-dev
Nodes: 1
Node: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.250)
Port: 9200 (default port)

[ OK ] 🚀 Auto mode/Default installation: Selecting Direct HTTPS configuration (option 1)

[ OK ] Configuring Direct HTTPS (Elasticsearch native SSL)...
──────────────────────────────────────────────────────────
[INFO] Environment: zone-universe-main-dev (1 nodes)
[INFO] Node: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.250)
[INFO] Port: 9200 (default port)
==================================================================
Direct HTTPS Configuration (native TLS, PEM)
==================================================================
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
Environment: zone-universe-main-dev
Nodes: 1
Node: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.250)
Port: 9200 (default port)
Domain: zone-universe-main-dev.fastorder.com

[INFO] === Single-Node Direct HTTPS Setup ===
🔐 Generated PKCS12 password: 8CbULBus... (32 chars)
[INFO] Using first node configuration: /etc/elasticsearch/zone-universe-main-dev/node-01
[2026-02-05 09:50:16 UTC] USER=www-data EUID=0 PID=613972 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/zone-universe-main-dev/node-01
[2026-02-05 09:50:16 UTC] USER=www-data EUID=0 PID=614007 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/zone-universe-main-dev/node-01
[INFO] Checking prerequisites...
[ OK ] ✓ Prerequisites verified
[INFO] Setting up temporary directories...
[2026-02-05 09:50:16 UTC] USER=www-data EUID=0 PID=614037 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs
[2026-02-05 09:50:16 UTC] USER=www-data EUID=0 PID=614046 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs
[2026-02-05 09:50:16 UTC] USER=www-data EUID=0 PID=614055 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/zone-universe-main-dev
[2026-02-05 09:50:16 UTC] USER=www-data EUID=0 PID=614065 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/zone-universe-main-dev/node-01
[2026-02-05 09:50:16 UTC] USER=www-data EUID=0 PID=614081 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs
[ OK ] ✓ Directories created

[INFO] Building certificate instances configuration...
[INFO] Certificate instances configuration:
instances:
  - name: "zone-universe-main-dev-http"
    dns:  [ "localhost", "web-03", "search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com", "zone-universe-main-dev-node-01.fastorder.com", "search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com", "search-zone-universe-main-dev.fastorder.com", "zone-universe-main-dev-node-01.local" ]
    ip:   [ "10.100.1.250", "127.0.0.1", "::1" ]
[ OK ] ✓ Instances configuration created

[INFO] Generating HTTP Certificate Authority...
[2026-02-05 09:50:16 UTC] USER=www-data EUID=0 PID=614121 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs
[2026-02-05 09:50:16 UTC] USER=www-data EUID=0 PID=614130 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs
[2026-02-05 09:50:22 UTC] USER=www-data EUID=0 PID=614335 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/http-ca.zip
Archive:  /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/http-ca.zip
  inflating: /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/ca/ca.crt  
  inflating: /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/ca/ca.key  
[ OK ] ✓ HTTP CA generated successfully

[INFO] Generating per-node HTTP certificates...
[2026-02-05 09:50:22 UTC] USER=www-data EUID=0 PID=614347 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out
[2026-02-05 09:50:22 UTC] USER=www-data EUID=0 PID=614356 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out
[2026-02-05 09:50:27 UTC] USER=www-data EUID=0 PID=614616 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out/http-certs.zip
[2026-02-05 09:50:27 UTC] USER=www-data EUID=0 PID=614625 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out/http
Archive:  /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out/http-certs.zip
  inflating: /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out/http/zone-universe-main-dev-http/zone-universe-main-dev-http.crt  
  inflating: /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out/http/zone-universe-main-dev-http/zone-universe-main-dev-http.key  
[ OK ] ✓ HTTP certificates generated successfully

[INFO] Installing certificates and configuring services...
[INFO] Configuring main service for single-node HTTPS...
[2026-02-05 09:50:27 UTC] USER=www-data EUID=0 PID=614647 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/zone-universe-main-dev/node-01/certs
[2026-02-05 09:50:27 UTC] USER=www-data EUID=0 PID=614656 ACTION=fsop ARGS=cp /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/out/http/zone-universe-main-dev-http/zone-universe-main-dev-http.crt /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http.crt
[2026-02-05 09:50:27 UTC] USER=www-data EUID=0 PID=614705 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/zone-universe-main-dev/node-01/certs
[2026-02-05 09:50:28 UTC] USER=www-data EUID=0 PID=614718 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http.key
[2026-02-05 09:50:28 UTC] USER=www-data EUID=0 PID=614727 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http.crt /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http_ca.crt /etc/elasticsearch/zone-universe-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-02-05 09:50:28 UTC] USER=www-data EUID=0 PID=614751 ACTION=fsop ARGS=sed -i -E -e /^\s*xpack\.security\.http\.ssl(\..*)?\s*:/d /etc/elasticsearch/zone-universe-main-dev/node-01/elasticsearch.yml
[2026-02-05 09:50:28 UTC] USER=www-data EUID=0 PID=614762 ACTION=fsop ARGS=sed -i /^# --- BEGIN direct HTTPS (managed, PEM) ---$/,/^# --- END direct HTTPS (managed, PEM) ---$/d /etc/elasticsearch/zone-universe-main-dev/node-01/elasticsearch.yml
[2026-02-05 09:50:28 UTC] USER=www-data EUID=0 PID=614771 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/zone-universe-main-dev/node-01/certs/fastorder_ra_root.crt
YAML: /etc/elasticsearch/zone-universe-main-dev/node-01/elasticsearch.yml
[ OK ]   ✓ Main service configured with HTTPS
[2026-02-05 09:50:28 UTC] USER=www-data EUID=0 PID=614796 ACTION=fsop ARGS=rm -rf /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients
[2026-02-05 09:50:28 UTC] USER=www-data EUID=0 PID=614805 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients
[2026-02-05 09:50:28 UTC] USER=www-data EUID=0 PID=614814 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients
Archive:  /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client.zip
   creating: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/
  inflating: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.crt  
  inflating: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.key  
[INFO] Creating P12 keystore for es-client...
[2026-02-05 09:50:33 UTC] USER=www-data EUID=0 PID=615002 ACTION=fsop ARGS=mv /tmp/es-client-613886.p12 /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[ OK ] ✓ Created P12 keystore: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-02-05 09:50:33 UTC] USER=www-data EUID=0 PID=615011 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients
[2026-02-05 09:50:33 UTC] USER=www-data EUID=0 PID=615020 ACTION=fsop ARGS=chmod 640 /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.key
[2026-02-05 09:50:33 UTC] USER=www-data EUID=0 PID=615029 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-02-05 09:50:33 UTC] USER=www-data EUID=0 PID=615041 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.crt /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http_ca.crt
[INFO] Saving keystore passwords to secrets vault...
🔑 Configuring AWS credentials for secrets vault...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[INFO] 🔐 Vaulting search passwords to remote backend...
✅ Passwords vaulted to remote backend
✓ Keystore passwords saved to secrets vault: search/zone-universe-main-dev/keystore-passwords

[INFO] === Installing CA Certificate for Users ===
[INFO] HOME not set, skipping user CA installation

✓ Direct HTTPS configuration completed for environment: zone-universe-main-dev
[INFO] All services now serve HTTPS using PEM certificates
[INFO] Network binding: 10.100.1.250
[INFO] HTTPS endpoint: https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === Certificate Summary ===
CA Certificate: /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs/ca/ca.crt
Certificate Directory: /etc/elasticsearch/zone-universe-main-dev/node-01/http-certs
Main service certificates: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/

[INFO] === Next Steps ===
1. Restart Elasticsearch services to apply HTTPS configuration
2. Test HTTPS connectivity: curl https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
3. Verify certificates: openssl s_client -connect search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === SSL Certificate Information ===
• CA certificate installed for user - curl will work without -k flag
• Each certificate includes node-specific domain, VM IP, and localhost in Subject Alternative Names
• Certificates are valid for 3 years from generation date

[WARNING] Important: You'll need to restart Elasticsearch services for HTTPS to take effect
[ OK ] ✓ Direct HTTPS configuration completed successfully

[ OK ] ==================================================================
[ OK ] HTTP SSL Configuration Complete
[ OK ] ==================================================================
[INFO] Environment: zone-universe-main-dev
[INFO] Nodes: 1
[INFO] Configuration applied to port: 9200 (default port for all nodes)

[INFO] === Next Steps ===
1. Verify Elasticsearch is running: systemctl status elasticsearch@zone-universe-main-dev-node-01.service
2. Test cluster health: curl https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health
3. Check SSL certificate: openssl s_client -connect search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
=== HTTPS Setup completed successfully! ===
Environment:  (1 nodes)
Domain: .fastorder.com
HTTPS endpoint: https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
Node IP: 10.100.1.250

✓ Step 2 completed successfully!

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 3: Executing Create Index Llm
Folder: 03-create-index-llm
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

==================================================================
Elasticsearch LLM/Semantic Search Setup
==================================================================
[INFO] Using web-provided environment: zone-universe-main-dev
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: zone-universe-main-dev
Service    : zone
🔍 Checking Elasticsearch availability…
✅ Elasticsearch is accessible at https://search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

=== Phase 1: Common steps under /data/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps ===
   (no numbered steps in: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps)
=== Phase 2: Service-scoped steps for 'zone' under /data/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/zone ===
⚠️  No service directory found for 'zone' at: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/zone
    Skipping service-scoped steps.
=== Phase 3: Optional search smoke tests ===
   (semantic search test script not found: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/search-semantic.sh)
   (hybrid search test script not found: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/hybrid-search.sh)
==================================================================
🎉 LLM/Semantic Search setup completed successfully!
==================================================================

Available commands:
  • Test semantic search:
    bash steps/search-semantic.sh en "password policy"
    bash steps/search-semantic.sh ar "كلمة المرور"

  • Test hybrid search:
    bash steps/hybrid-search.sh en "user authentication"
    bash steps/hybrid-search.sh ar "مصادقة المستخدم"

Alias   : zone_universe_main_dev_account_router
Index   : zone_universe_main_dev_account_router-000001
ILM     : zone-account-router-ilm
Model   : zone-text-embedding-001
Pipeline: zone-embed-pipeline-001
==================================================================

✓ Step 3 completed successfully!

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 4: Executing Monitoring Setup
Folder: 10-monitoring-setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using web-provided environment: zone-universe-main-dev
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Elasticsearch Monitoring Integration for zone-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-zone-universe-main-dev
[OK]   Observability cell endpoints registered for zone-universe-main-dev
[OK]   ✓ Observability cell is ready

[INFO] 2️⃣ Discovering Elasticsearch configuration...
[OK]   ✓ Found Elasticsearch at 10.100.1.250:9200

[INFO] 3️⃣ Setting up elasticsearch_exporter integration...
[INFO] Using elasticsearch_exporter port: 9114
[INFO] SSL certificates configured for elasticsearch_exporter:
[INFO]   CA cert: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/http_ca.crt
[INFO]   Client cert: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.crt
[INFO]   Client key: /etc/elasticsearch/zone-universe-main-dev/node-01/certs/clients/es-client/es-client.key
[INFO] Checking observability cell readiness: obs-zone-universe-main-dev
[OK]   Observability cell endpoints registered for zone-universe-main-dev
[INFO] Setting up elasticsearch_exporter for zone-universe-main-dev
[INFO] Elasticsearch exporter will bind to: 10.100.1.250:9114
[2026-02-05 09:50:43 UTC] USER=www-data EUID=0 PID=615564 ACTION=passthru ARGS=mv /tmp/elasticsearch_exporter-zone-universe-main-dev.service /etc/systemd/system/elasticsearch_exporter-zone-universe-main-dev.service
[2026-02-05 09:50:43 UTC] USER=www-data EUID=0 PID=615573 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 09:50:43 UTC] USER=www-data EUID=0 PID=615622 ACTION=passthru ARGS=systemctl enable elasticsearch_exporter-zone-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/elasticsearch_exporter-zone-universe-main-dev.service → /etc/systemd/system/elasticsearch_exporter-zone-universe-main-dev.service.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  IP Conflict Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: zone-universe-main-dev
IP Address:  10.100.1.250
Port:        9114
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔍 Checking IP conflict for zone-universe-main-dev on 10.100.1.250:9114...
✅ IP 10.100.1.250:9114 is available - no conflicts detected

🔍 Checking for orphaned processes that might conflict...
✅ No orphaned processes detected

✅ All checks passed - safe to proceed with zone-universe-main-dev setup
[2026-02-05 09:50:44 UTC] USER=www-data EUID=0 PID=615747 ACTION=passthru ARGS=systemctl restart elasticsearch_exporter-zone-universe-main-dev.service
[OK]   elasticsearch_exporter configured on 10.100.1.250:9114
[INFO] Register this endpoint in metrics-zone-universe-main-dev.fastorder.com scrape config
[OK]   ✓ elasticsearch_exporter integration complete

[INFO] 3.5️⃣ Configuring Prometheus to scrape Elasticsearch metrics...
[2026-02-05 09:50:47 UTC] USER=www-data EUID=0 PID=615869 ACTION=passthru ARGS=grep -q job_name: 'elasticsearch' /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml
[INFO] Adding Elasticsearch scrape target to Prometheus configuration...
[2026-02-05 09:50:47 UTC] USER=www-data EUID=0 PID=615892 ACTION=fsop ARGS=cp /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml.backup-1770285047
[INFO] Created backup: /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml.backup-1770285047
[2026-02-05 09:50:47 UTC] USER=www-data EUID=0 PID=615934 ACTION=passthru ARGS=grep -q job_name: 'elasticsearch' /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml
[INFO] ✓ Elasticsearch job successfully inserted into config
[INFO] Validating Prometheus configuration with promtool...
Checking /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: /etc/prometheus/obs-zone-universe-main-dev/prometheus.yml is valid prometheus config file syntax

Checking /etc/prometheus/obs-zone-universe-main-dev/rules/basic_alerts.yml
  SUCCESS: 4 rules found

[OK]   ✓ Prometheus configuration validation PASSED
[OK]   ✓ Prometheus configuration updated successfully
[2026-02-05 09:50:47 UTC] USER=www-data EUID=0 PID=615964 ACTION=passthru ARGS=systemctl is-active --quiet prometheus-obs-zone-universe-main-dev.service
[INFO] Reloading Prometheus configuration...
[2026-02-05 09:50:47 UTC] USER=www-data EUID=0 PID=615985 ACTION=passthru ARGS=systemctl restart prometheus-obs-zone-universe-main-dev.service
[2026-02-05 09:50:50 UTC] USER=www-data EUID=0 PID=616173 ACTION=passthru ARGS=systemctl is-active --quiet prometheus-obs-zone-universe-main-dev.service
[OK]   ✓ Prometheus reloaded successfully
[2026-02-05 09:50:50 UTC] USER=www-data EUID=0 PID=616202 ACTION=fsop ARGS=rm -f /tmp/prometheus_es_add.yml

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Elasticsearch Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Service: elasticsearch_exporter-zone-universe-main-dev.service
[INFO] Metrics: http://localhost:9114/metrics
[INFO] Prometheus: https://metrics-zone-universe-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-zone-universe-main-dev.fastorder.com
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 4️⃣ Registering Elasticsearch nodes to monitoring database...
[INFO]    Constructed FQDN: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com
[INFO] Registering: zone-universe-main-dev-node-01
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Elasticsearch
[INFO]   Identifier:        zone-universe-main-dev-node-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.250
[INFO]   Port:              9200
[INFO]   FQDN:              search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       zone-universe-main-dev (service=zone, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: df7ce767-601b-4c18-81af-9d78f9ffd060
[SUCCESS] Environment UUID: 51cbf631-2683-474f-9770-5018428c13a0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/51cbf631-2683-474f-9770-5018428c13a0
[OK]   ✓ Registered: zone-universe-main-dev-node-01
[OK]   ✓ Elasticsearch node registration completed successfully

[INFO] 5️⃣ Verifying monitoring integration...

[INFO] Checking elasticsearch_exporter service...
[OK]   ✓ elasticsearch_exporter-zone-universe-main-dev.service is ACTIVE
[INFO] Checking Prometheus service...
[OK]   ✓ prometheus-obs-zone-universe-main-dev.service is ACTIVE
[INFO] Validating Prometheus configuration...
[OK]   ✓ Prometheus configuration is VALID
[INFO] Checking Prometheus targets (waiting 35s for first scrape cycle)...
[2026-02-05 09:51:26 UTC] USER=www-data EUID=0 PID=617747 ACTION=passthru ARGS=grep -q tls_server_config /etc/prometheus/obs-zone-universe-main-dev/web-config.yml
[OK]   ✓ Prometheus has Elasticsearch target configured
[OK]   ✓ Elasticsearch target is UP and being scraped

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ All monitoring integration steps completed
[INFO] ✅ All verifications PASSED
[INFO] ✅ Elasticsearch registered to dashboard database
[INFO] ✅ Prometheus scraping Elasticsearch metrics
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


✓ Step 4 completed successfully!

════════════════════════════════════════════════════════════════
🎉 All deployment tasks completed successfully!

✓ ✅ Search infrastructure (elasticsearch) setup completed successfully