📊 Provisioning Job Status

Environment: Zone Universe Main Dev on web-03

❌ Failed

⏱️ Timing Summary

🕐

Requested 2026-02-05 09:24:56 2 days ago

▶️

Started 2026-02-05 09:24:56 2 days ago

🏁

Finished 2026-02-05 09:24:57 2 days ago

⏲️

Total Duration 1 second

📋 Job Details

Job ID: c620d252-a1d3-47d9-90d7-c16d1e75552b

Action: SETUP

Status: ❌ FAILED

Environment: zone-universe-main-dev

Resource: web-03 (Provider)

Requested By: admin

Parameters:

"{\"env\": \"dev\", \"zone\": \"universe\", \"branch\": \"main\", \"db_app\": \"postgresql\", \"service\": \"zone\", \"es_nodes\": 1, \"db_enabled\": true, \"pg_standby\": 1, \"pg_workers\": 1, \"search_app\": \"elasticsearch\", \"description\": \"\", \"iam_enabled\": false, \"worker_1_ip\": \"10.100.1.42\", \"eventbus_app\": \"kafka\", \"es_https_mode\": \"direct\", \"service_es_ip\": \"10.100.1.4\", \"worker_1_fqdn\": \"db-zone-universe-main-dev-postgresql-worker-01.fastorder.com\", \"search_enabled\": true, \"service_app_ip\": \"10.100.1.2\", \"service_obs_ip\": \"10.100.1.18\", \"service_es_fqdn\": \"search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com\", \"service_otlp_ip\": \"10.100.1.30\", \"eventbus_enabled\": true, \"service_app_fqdn\": \"app-zone-universe-main-dev.fastorder.com\", \"service_audit_ip\": \"10.100.1.32\", \"service_obs_fqdn\": \"obs-zone-universe-main-dev.fastorder.com\", \"service_tempo_ip\": \"10.100.1.28\", \"service_endpoints\": \"[{\\\"ip\\\":\\\"10.100.1.3\\\",\\\"fqdn\\\":\\\"app-zone-universe-main-dev.fastorder.com\\\",\\\"service\\\":\\\"app\\\"},{\\\"ip\\\":\\\"10.100.1.5\\\",\\\"fqdn\\\":\\\"search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com\\\",\\\"service\\\":\\\"es_coordinator\\\"},{\\\"ip\\\":\\\"10.100.1.7\\\",\\\"fqdn\\\":\\\"search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com\\\",\\\"service\\\":\\\"es_node_1\\\"},{\\\"ip\\\":\\\"10.100.1.9\\\",\\\"fqdn\\\":\\\"eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com\\\",\\\"service\\\":\\\"kafka_broker_1\\\"},{\\\"ip\\\":\\\"10.100.1.11\\\",\\\"fqdn\\\":\\\"eventbus-zone-universe-main-dev-kafka-connect.fastorder.com\\\",\\\"service\\\":\\\"kafka_connect\\\"},{\\\"ip\\\":\\\"10.100.1.13\\\",\\\"fqdn\\\":\\\"schema-zone-universe-main-dev-kafka-registry.fastorder.com\\\",\\\"service\\\":\\\"kafka_registry\\\"},{\\\"ip\\\":\\\"10.100.1.15\\\",\\\"fqdn\\\":\\\"db-zone-universe-main-dev-postgresql-coordinator.fastorder.com\\\",\\\"service\\\":\\\"pg_coordinator\\\"},{\\\"ip\\\":\\\"10.100.1.17\\\",\\\"fqdn\\\":\\\"db-zone-universe-main-dev-postgresql-bouncer.fastorder.com\\\",\\\"service\\\":\\\"pgbouncer\\\"},{\\\"ip\\\":\\\"10.100.1.19\\\",\\\"fqdn\\\":\\\"obs-zone-universe-main-dev.fastorder.com\\\",\\\"service\\\":\\\"obs\\\"},{\\\"ip\\\":\\\"10.100.1.21\\\",\\\"fqdn\\\":\\\"metrics-zone-universe-main-dev-prometheus.fastorder.com\\\",\\\"service\\\":\\\"metrics\\\"},{\\\"ip\\\":\\\"10.100.1.23\\\",\\\"fqdn\\\":\\\"dashboards-zone-universe-main-dev-grafana.fastorder.com\\\",\\\"service\\\":\\\"dashboards\\\"},{\\\"ip\\\":\\\"10.100.1.25\\\",\\\"fqdn\\\":\\\"alerts-zone-universe-main-dev-alertmanager.fastorder.com\\\",\\\"service\\\":\\\"alerts\\\"},{\\\"ip\\\":\\\"10.100.1.27\\\",\\\"fqdn\\\":\\\"logstore-zone-universe-main-dev-clickhouse.fastorder.com\\\",\\\"service\\\":\\\"logs\\\"},{\\\"ip\\\":\\\"10.100.1.29\\\",\\\"fqdn\\\":\\\"traces-zone-universe-main-dev-tempo.fastorder.com\\\",\\\"service\\\":\\\"traces\\\"},{\\\"ip\\\":\\\"10.100.1.31\\\",\\\"fqdn\\\":\\\"telemetry-zone-universe-main-dev-opentelemetry.fastorder.com\\\",\\\"service\\\":\\\"telemetry\\\"},{\\\"ip\\\":\\\"10.100.1.33\\\",\\\"fqdn\\\":\\\"audit-zone-universe-main-dev.fastorder.com\\\",\\\"service\\\":\\\"audit\\\"},{\\\"ip\\\":\\\"10.100.1.35\\\",\\\"fqdn\\\":\\\"backup-zone-universe-main-dev-db-postgresql.fastorder.com\\\",\\\"service\\\":\\\"backup_pg\\\"},{\\\"ip\\\":\\\"10.100.1.37\\\",\\\"fqdn\\\":\\\"backup-zone-universe-main-dev-eventbus-kafka.fastorder.com\\\",\\\"service\\\":\\\"backup_kafka\\\"},{\\\"ip\\\":\\\"10.100.1.39\\\",\\\"fqdn\\\":\\\"backup-zone-universe-main-dev-search-elasticsearch.fastorder.com\\\",\\\"service\\\":\\\"backup_es\\\"},{\\\"ip\\\":\\\"10.100.1.41\\\",\\\"fqdn\\\":\\\"backup-zone-universe-main-dev-orchestrator.fastorder.com\\\",\\\"service\\\":\\\"backup_orchestrator\\\"}]\", \"service_otlp_fqdn\": \"telemetry-zone-universe-main-dev-opentelemetry.fastorder.com\", \"postgresql_enabled\": true, \"service_audit_fqdn\": \"audit-zone-universe-main-dev.fastorder.com\", \"service_grafana_ip\": \"10.100.1.22\", \"service_tempo_fqdn\": \"traces-zone-universe-main-dev-tempo.fastorder.com\", \"service_backup_es_ip\": \"10.100.1.38\", \"service_backup_pg_ip\": \"10.100.1.34\", \"service_es_node_1_ip\": \"10.100.1.6\", \"service_grafana_fqdn\": \"dashboards-zone-universe-main-dev-grafana.fastorder.com\", \"service_pgbouncer_ip\": \"10.100.1.16\", \"service_prometheus_ip\": \"10.100.1.20\", \"worker_1_standby_1_ip\": \"10.100.1.43\", \"service_backup_es_fqdn\": \"backup-zone-universe-main-dev-search-elasticsearch.fastorder.com\", \"service_backup_pg_fqdn\": \"backup-zone-universe-main-dev-db-postgresql.fastorder.com\", \"service_es_node_1_fqdn\": \"search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com\", \"service_log_backend_ip\": \"10.100.1.26\", \"service_pgbouncer_fqdn\": \"db-zone-universe-main-dev-postgresql-bouncer.fastorder.com\", \"service_alertmanager_ip\": \"10.100.1.24\", \"service_backup_kafka_ip\": \"10.100.1.36\", \"service_prometheus_fqdn\": \"metrics-zone-universe-main-dev-prometheus.fastorder.com\", \"worker_1_standby_1_fqdn\": \"db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com\", \"service_kafka_connect_ip\": \"10.100.1.10\", \"service_log_backend_fqdn\": \"logstore-zone-universe-main-dev-clickhouse.fastorder.com\", \"service_alertmanager_fqdn\": \"alerts-zone-universe-main-dev-alertmanager.fastorder.com\", \"service_backup_kafka_fqdn\": \"backup-zone-universe-main-dev-eventbus-kafka.fastorder.com\", \"service_kafka_broker_1_ip\": \"10.100.1.8\", \"service_kafka_registry_ip\": \"10.100.1.12\", \"service_pg_coordinator_ip\": \"10.100.1.14\", \"service_kafka_connect_fqdn\": \"eventbus-zone-universe-main-dev-kafka-connect.fastorder.com\", \"postgresql_run_verification\": true, \"service_kafka_broker_1_fqdn\": \"eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com\", \"service_kafka_registry_fqdn\": \"schema-zone-universe-main-dev-kafka-registry.fastorder.com\", \"service_pg_coordinator_fqdn\": \"db-zone-universe-main-dev-postgresql-coordinator.fastorder.com\", \"service_backup_orchestrator_ip\": \"10.100.1.40\", \"service_backup_orchestrator_fqdn\": \"backup-zone-universe-main-dev-orchestrator.fastorder.com\"}"

❌ Error: One or more steps failed. Check run logs for details.

⚠️ Job Failed

This job encountered an error. You can restart from the failed step.

📢 Viewing Old Job Attempt

This job has been restarted. You are viewing an older attempt. The logs and status shown below are from the latest retry.

🔄 View Latest Attempt Latest Status:

🔄 Resume & Restart Options

This job failed at one of the steps below. You can resume from where it failed to save time and avoid re-running successful steps.

💡

1 step failed

📝 Execution Steps (9)

0/9 completed 1 failed

0% (0/9 steps)

00-preflight-checks local

❌ FAILED

⏰ Started: 2026-02-05 09:24:56

🏁 Finished: 2026-02-05 09:24:57

⏱️ Duration: 1 second

📄 View Logs (2279 chars)


[1m════════════════════════════════════════════════════════════════[0m
[1m  FastOrder Pre-Flight Validation Checks[0m
[1m════════════════════════════════════════════════════════════════[0m

[0;34m[INFO][0m Checking SSH connectivity to target host...
[0;32m[✓][0m Target is localhost, skipping SSH check

[0;34m[INFO][0m Checking available disk space...
[0;34m[INFO][0m Checking /data disk (mounted separately for data storage)
[0;32m[✓][0m Disk space sufficient: 287GB available (required: 50GB)

[0;34m[INFO][0m Checking available memory...
[1;33m[⚠][0m Memory limited: 15GB (recommended: 16GB)
  → Consider reducing Elasticsearch nodes or PostgreSQL workers

[0;34m[INFO][0m Checking critical port availability...
[0;32m[✓][0m Port 5432 in use on specific IP (10.100.1.189:5432) - OK, can use different IP
[0;32m[✓][0m Port 9200 in use on specific IP ([::ffff:10.100.1.179]) - OK, can use different IP
[0;32m[✓][0m Port 9300 in use on specific IP ([::ffff:10.100.1.186]) - OK, can use different IP
[0;32m[✓][0m Port 9092 in use on specific IP ([::ffff:10.100.1.225]) - OK, can use different IP
[0;32m[✓][0m Port 2181 available (Zookeeper)

[0;34m[INFO][0m Checking DNS resolution...
[0;32m[✓][0m DNS resolution working: google.com
[0;32m[✓][0m DNS resolution working: github.com
[0;32m[✓][0m DNS resolution working: archive.ubuntu.com

[0;34m[INFO][0m Checking required system commands...
[0;32m[✓][0m Command available: curl
[0;32m[✓][0m Command available: wget
[0;32m[✓][0m Command available: git
[0;32m[✓][0m Command available: sudo
[0;32m[✓][0m Command available: systemctl
[0;32m[✓][0m Command available: apt-get

[0;34m[INFO][0m Checking current system load...
[0;31m[✗][0m System load very high: 6.04 (4 CPUs)
  → Wait for system to stabilize before provisioning

[0;34m[INFO][0m Checking for existing environment conflicts...
[0;32m[✓][0m No conflicting services found for: zone-uae-main-dev

[1m════════════════════════════════════════════════════════════════[0m
[1m  Pre-Flight Check Summary[0m
[1m════════════════════════════════════════════════════════════════[0m
[0;31m[✗][0m 1 error(s) detected, 1 warning(s)

❌ Environment NOT ready for provisioning
   Fix critical errors before proceeding

📥 Download Full Logs

00-terraform-provision local

⏸️ PENDING