πŸ“Š Provisioning Job Status

Environment: Zone Universe Main Dev on web-03

❌ Failed

⏱️ Timing Summary

πŸ•
Requested 2026-02-05 09:27:28 2 days ago
▢️
Started 2026-02-05 09:27:28 2 days ago
🏁
Finished 2026-02-05 09:28:12 2 days ago
⏲️
Total Duration 44 seconds

πŸ“‹ Job Details

Job ID: b57e3152-ce34-47b2-b9a3-230702bc4338
Action: SETUP
Status: ❌ FAILED
Environment: zone-universe-main-dev
Resource: web-03 (Provider)
Requested By: admin
Parameters:
"{\"env\": \"dev\", \"zone\": \"universe\", \"branch\": \"main\", \"db_app\": \"postgresql\", \"service\": \"zone\", \"es_nodes\": 1, \"db_enabled\": true, \"pg_standby\": 1, \"pg_workers\": 1, \"search_app\": \"elasticsearch\", \"description\": \"\", \"iam_enabled\": false, \"worker_1_ip\": \"10.100.1.42\", \"eventbus_app\": \"kafka\", \"es_https_mode\": \"direct\", \"service_es_ip\": \"10.100.1.4\", \"worker_1_fqdn\": \"db-zone-universe-main-dev-postgresql-worker-01.fastorder.com\", \"search_enabled\": true, \"service_app_ip\": \"10.100.1.2\", \"service_obs_ip\": \"10.100.1.18\", \"service_es_fqdn\": \"search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com\", \"service_otlp_ip\": \"10.100.1.30\", \"eventbus_enabled\": true, \"service_app_fqdn\": \"app-zone-universe-main-dev.fastorder.com\", \"service_audit_ip\": \"10.100.1.32\", \"service_obs_fqdn\": \"obs-zone-universe-main-dev.fastorder.com\", \"service_tempo_ip\": \"10.100.1.28\", \"service_endpoints\": \"[{\\\"ip\\\":\\\"10.100.1.3\\\",\\\"fqdn\\\":\\\"app-zone-universe-main-dev.fastorder.com\\\",\\\"service\\\":\\\"app\\\"},{\\\"ip\\\":\\\"10.100.1.5\\\",\\\"fqdn\\\":\\\"search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com\\\",\\\"service\\\":\\\"es_coordinator\\\"},{\\\"ip\\\":\\\"10.100.1.7\\\",\\\"fqdn\\\":\\\"search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com\\\",\\\"service\\\":\\\"es_node_1\\\"},{\\\"ip\\\":\\\"10.100.1.9\\\",\\\"fqdn\\\":\\\"eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com\\\",\\\"service\\\":\\\"kafka_broker_1\\\"},{\\\"ip\\\":\\\"10.100.1.11\\\",\\\"fqdn\\\":\\\"eventbus-zone-universe-main-dev-kafka-connect.fastorder.com\\\",\\\"service\\\":\\\"kafka_connect\\\"},{\\\"ip\\\":\\\"10.100.1.13\\\",\\\"fqdn\\\":\\\"schema-zone-universe-main-dev-kafka-registry.fastorder.com\\\",\\\"service\\\":\\\"kafka_registry\\\"},{\\\"ip\\\":\\\"10.100.1.15\\\",\\\"fqdn\\\":\\\"db-zone-universe-main-dev-postgresql-coordinator.fastorder.com\\\",\\\"service\\\":\\\"pg_coordinator\\\"},{\\\"ip\\\":\\\"10.100.1.17\\\",\\\"fqdn\\\":\\\"db-zone-universe-main-dev-postgresql-bouncer.fastorder.com\\\",\\\"service\\\":\\\"pgbouncer\\\"},{\\\"ip\\\":\\\"10.100.1.19\\\",\\\"fqdn\\\":\\\"obs-zone-universe-main-dev.fastorder.com\\\",\\\"service\\\":\\\"obs\\\"},{\\\"ip\\\":\\\"10.100.1.21\\\",\\\"fqdn\\\":\\\"metrics-zone-universe-main-dev-prometheus.fastorder.com\\\",\\\"service\\\":\\\"metrics\\\"},{\\\"ip\\\":\\\"10.100.1.23\\\",\\\"fqdn\\\":\\\"dashboards-zone-universe-main-dev-grafana.fastorder.com\\\",\\\"service\\\":\\\"dashboards\\\"},{\\\"ip\\\":\\\"10.100.1.25\\\",\\\"fqdn\\\":\\\"alerts-zone-universe-main-dev-alertmanager.fastorder.com\\\",\\\"service\\\":\\\"alerts\\\"},{\\\"ip\\\":\\\"10.100.1.27\\\",\\\"fqdn\\\":\\\"logstore-zone-universe-main-dev-clickhouse.fastorder.com\\\",\\\"service\\\":\\\"logs\\\"},{\\\"ip\\\":\\\"10.100.1.29\\\",\\\"fqdn\\\":\\\"traces-zone-universe-main-dev-tempo.fastorder.com\\\",\\\"service\\\":\\\"traces\\\"},{\\\"ip\\\":\\\"10.100.1.31\\\",\\\"fqdn\\\":\\\"telemetry-zone-universe-main-dev-opentelemetry.fastorder.com\\\",\\\"service\\\":\\\"telemetry\\\"},{\\\"ip\\\":\\\"10.100.1.33\\\",\\\"fqdn\\\":\\\"audit-zone-universe-main-dev.fastorder.com\\\",\\\"service\\\":\\\"audit\\\"},{\\\"ip\\\":\\\"10.100.1.35\\\",\\\"fqdn\\\":\\\"backup-zone-universe-main-dev-db-postgresql.fastorder.com\\\",\\\"service\\\":\\\"backup_pg\\\"},{\\\"ip\\\":\\\"10.100.1.37\\\",\\\"fqdn\\\":\\\"backup-zone-universe-main-dev-eventbus-kafka.fastorder.com\\\",\\\"service\\\":\\\"backup_kafka\\\"},{\\\"ip\\\":\\\"10.100.1.39\\\",\\\"fqdn\\\":\\\"backup-zone-universe-main-dev-search-elasticsearch.fastorder.com\\\",\\\"service\\\":\\\"backup_es\\\"},{\\\"ip\\\":\\\"10.100.1.41\\\",\\\"fqdn\\\":\\\"backup-zone-universe-main-dev-orchestrator.fastorder.com\\\",\\\"service\\\":\\\"backup_orchestrator\\\"}]\", \"service_otlp_fqdn\": \"telemetry-zone-universe-main-dev-opentelemetry.fastorder.com\", \"postgresql_enabled\": true, \"service_audit_fqdn\": \"audit-zone-universe-main-dev.fastorder.com\", \"service_grafana_ip\": \"10.100.1.22\", \"service_tempo_fqdn\": \"traces-zone-universe-main-dev-tempo.fastorder.com\", \"service_backup_es_ip\": \"10.100.1.38\", \"service_backup_pg_ip\": \"10.100.1.34\", \"service_es_node_1_ip\": \"10.100.1.6\", \"service_grafana_fqdn\": \"dashboards-zone-universe-main-dev-grafana.fastorder.com\", \"service_pgbouncer_ip\": \"10.100.1.16\", \"service_prometheus_ip\": \"10.100.1.20\", \"worker_1_standby_1_ip\": \"10.100.1.43\", \"service_backup_es_fqdn\": \"backup-zone-universe-main-dev-search-elasticsearch.fastorder.com\", \"service_backup_pg_fqdn\": \"backup-zone-universe-main-dev-db-postgresql.fastorder.com\", \"service_es_node_1_fqdn\": \"search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com\", \"service_log_backend_ip\": \"10.100.1.26\", \"service_pgbouncer_fqdn\": \"db-zone-universe-main-dev-postgresql-bouncer.fastorder.com\", \"service_alertmanager_ip\": \"10.100.1.24\", \"service_backup_kafka_ip\": \"10.100.1.36\", \"service_prometheus_fqdn\": \"metrics-zone-universe-main-dev-prometheus.fastorder.com\", \"worker_1_standby_1_fqdn\": \"db-zone-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com\", \"service_kafka_connect_ip\": \"10.100.1.10\", \"service_log_backend_fqdn\": \"logstore-zone-universe-main-dev-clickhouse.fastorder.com\", \"service_alertmanager_fqdn\": \"alerts-zone-universe-main-dev-alertmanager.fastorder.com\", \"service_backup_kafka_fqdn\": \"backup-zone-universe-main-dev-eventbus-kafka.fastorder.com\", \"service_kafka_broker_1_ip\": \"10.100.1.8\", \"service_kafka_registry_ip\": \"10.100.1.12\", \"service_pg_coordinator_ip\": \"10.100.1.14\", \"service_kafka_connect_fqdn\": \"eventbus-zone-universe-main-dev-kafka-connect.fastorder.com\", \"postgresql_run_verification\": true, \"service_kafka_broker_1_fqdn\": \"eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com\", \"service_kafka_registry_fqdn\": \"schema-zone-universe-main-dev-kafka-registry.fastorder.com\", \"service_pg_coordinator_fqdn\": \"db-zone-universe-main-dev-postgresql-coordinator.fastorder.com\", \"service_backup_orchestrator_ip\": \"10.100.1.40\", \"service_backup_orchestrator_fqdn\": \"backup-zone-universe-main-dev-orchestrator.fastorder.com\"}"
❌ Error: One or more steps failed. Check run logs for details.
⚠️ Job Failed

This job encountered an error. You can restart from the failed step.

πŸ“’ Viewing Old Job Attempt

This job has been restarted. You are viewing an older attempt. The logs and status shown below are from the latest retry.

πŸ”„ Resume & Restart Options

This job failed at one of the steps below. You can resume from where it failed to save time and avoid re-running successful steps.

πŸ’‘
3 steps completed, 1 step failed
Completed steps will be skipped when you resume.

πŸ“ Execution Steps (9)

3/9 completed 1 failed
33% (3/9 steps)
1
00-preflight-checks local
βœ… SUCCEEDED
⏰ Started: 2026-02-05 09:27:28
🏁 Finished: 2026-02-05 09:27:29
⏱️ Duration: 1 second
πŸ“„ View Logs (2267 chars)

════════════════════════════════════════════════════════════════
  FastOrder Pre-Flight Validation Checks
════════════════════════════════════════════════════════════════

[INFO] Checking SSH connectivity to target host...
[βœ“] Target is localhost, skipping SSH check

[INFO] Checking available disk space...
[INFO] Checking /data disk (mounted separately for data storage)
[βœ“] Disk space sufficient: 287GB available (required: 50GB)

[INFO] Checking available memory...
[⚠] Memory limited: 15GB (recommended: 16GB)
  β†’ Consider reducing Elasticsearch nodes or PostgreSQL workers

[INFO] Checking critical port availability...
[βœ“] Port 5432 in use on specific IP (10.100.1.189:5432) - OK, can use different IP
[βœ“] Port 9200 in use on specific IP ([::ffff:10.100.1.179]) - OK, can use different IP
[βœ“] Port 9300 in use on specific IP ([::ffff:10.100.1.186]) - OK, can use different IP
[βœ“] Port 9092 in use on specific IP ([::ffff:10.100.1.225]) - OK, can use different IP
[βœ“] Port 2181 available (Zookeeper)

[INFO] Checking DNS resolution...
[βœ“] DNS resolution working: google.com
[βœ“] DNS resolution working: github.com
[βœ“] DNS resolution working: archive.ubuntu.com

[INFO] Checking required system commands...
[βœ“] Command available: curl
[βœ“] Command available: wget
[βœ“] Command available: git
[βœ“] Command available: sudo
[βœ“] Command available: systemctl
[βœ“] Command available: apt-get

[INFO] Checking current system load...
[⚠] System load elevated: 4.10 (4 CPUs)
  β†’ Provisioning may be slower than expected

[INFO] Checking for existing environment conflicts...
[βœ“] No conflicting services found for: zone-uae-main-dev

════════════════════════════════════════════════════════════════
  Pre-Flight Check Summary
════════════════════════════════════════════════════════════════
[⚠] 2 warning(s) detected

⚠️  Environment can proceed with caution
   Review warnings above and consider remediation
2
00-terraform-provision local
βœ… SUCCEEDED
⏰ Started: 2026-02-05 09:27:29
🏁 Finished: 2026-02-05 09:27:51
⏱️ Duration: 22 seconds
πŸ“„ View Logs (32163 chars)
[INFO] Using web-provided environment: zone-universe-main-dev
[INFO] Auto-creating state directory for zone-universe-main-dev...
[ OK ] Created topology.json for zone-universe-main-dev
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=10.100.1.51)
[2026-02-05_09:27:30] Starting Terraform provisioning step
[2026-02-05_09:27:30] Service: zone
[2026-02-05_09:27:30] Zone: universe
[2026-02-05_09:27:30] Environment: dev
[2026-02-05_09:27:30] Resource: web-03
[2026-02-05_09:27:30] Terraform binary: /home/ab/bin/terraform
[2026-02-05_09:27:30] HOME: /home/www-data
[2026-02-05_09:27:30] AWS Config: /home/ab/.aws/config
[2026-02-05_09:27:30] AWS Credentials: /home/ab/.aws/credentials
[2026-02-05_09:27:30] Terraform directory: /opt/fastorder/cli/terraform/examples/citus-production
[2026-02-05_09:27:30] Running terraform init...

Initializing the backend...
Upgrading modules...
- citus_cluster in ../../modules/citus_cluster

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 5.0"...
- Using previously-installed hashicorp/aws v5.100.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
[2026-02-05_09:27:36] βœ“ Terraform init succeeded
[2026-02-05_09:27:36] Running terraform validate...
Success! The configuration is valid.

[2026-02-05_09:27:41] βœ“ Terraform validate succeeded
[2026-02-05_09:27:41] Running terraform plan...
module.citus_cluster.data.aws_caller_identity.current: Reading...
module.citus_cluster.data.aws_caller_identity.current: Read complete after 0s [id=464621692046]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.citus_cluster.aws_iam_instance_profile.citus will be created
  + resource "aws_iam_instance_profile" "citus" {
      + arn         = (known after apply)
      + create_date = (known after apply)
      + id          = (known after apply)
      + name        = (known after apply)
      + name_prefix = "citus-prod-"
      + path        = "/"
      + role        = (known after apply)
      + tags        = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-prod"
        }
      + tags_all    = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
        }
      + unique_id   = (known after apply)
    }

  # module.citus_cluster.aws_iam_role.citus will be created
  + resource "aws_iam_role" "citus" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRole"
                      + Effect    = "Allow"
                      + Principal = {
                          + Service = "ec2.amazonaws.com"
                        }
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = (known after apply)
      + name_prefix           = "citus-prod-"
      + path                  = "/"
      + tags                  = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-prod"
        }
      + tags_all              = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
        }
      + unique_id             = (known after apply)
    }

  # module.citus_cluster.aws_iam_role_policy.secrets_manager[0] will be created
  + resource "aws_iam_role_policy" "secrets_manager" {
      + id          = (known after apply)
      + name        = (known after apply)
      + name_prefix = "secrets-access-"
      + policy      = jsonencode(
            {
              + Statement = [
                  + {
                      + Action   = [
                          + "secretsmanager:GetSecretValue",
                          + "secretsmanager:DescribeSecret",
                        ]
                      + Effect   = "Allow"
                      + Resource = "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/db/web/ksa/main/dev/postgresqladmin/ksa/prod*"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + role        = (known after apply)
    }

  # module.citus_cluster.aws_iam_role_policy_attachment.cloudwatch will be created
  + resource "aws_iam_role_policy_attachment" "cloudwatch" {
      + id         = (known after apply)
      + policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
      + role       = (known after apply)
    }

  # module.citus_cluster.aws_iam_role_policy_attachment.ssm will be created
  + resource "aws_iam_role_policy_attachment" "ssm" {
      + id         = (known after apply)
      + policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
      + role       = (known after apply)
    }

  # module.citus_cluster.aws_instance.coordinator will be created
  + resource "aws_instance" "coordinator" {
      + ami                                  = "ami-0b2aae5f4283c0df2"
      + arn                                  = (known after apply)
      + associate_public_ip_address          = (known after apply)
      + availability_zone                    = (known after apply)
      + cpu_core_count                       = (known after apply)
      + cpu_threads_per_core                 = (known after apply)
      + disable_api_stop                     = (known after apply)
      + disable_api_termination              = (known after apply)
      + ebs_optimized                        = (known after apply)
      + enable_primary_ipv6                  = (known after apply)
      + get_password_data                    = false
      + host_id                              = (known after apply)
      + host_resource_group_arn              = (known after apply)
      + iam_instance_profile                 = (known after apply)
      + id                                   = (known after apply)
      + instance_initiated_shutdown_behavior = (known after apply)
      + instance_lifecycle                   = (known after apply)
      + instance_state                       = (known after apply)
      + instance_type                        = "r6i.2xlarge"
      + ipv6_address_count                   = (known after apply)
      + ipv6_addresses                       = (known after apply)
      + key_name                             = (known after apply)
      + monitoring                           = (known after apply)
      + outpost_arn                          = (known after apply)
      + password_data                        = (known after apply)
      + placement_group                      = (known after apply)
      + placement_partition_number           = (known after apply)
      + primary_network_interface_id         = (known after apply)
      + private_dns                          = (known after apply)
      + private_ip                           = (known after apply)
      + public_dns                           = (known after apply)
      + public_ip                            = (known after apply)
      + secondary_private_ips                = (known after apply)
      + security_groups                      = (known after apply)
      + source_dest_check                    = true
      + spot_instance_request_id             = (known after apply)
      + subnet_id                            = "subnet-0a1f5a9a74ed030cf"
      + tags                                 = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-coordinator-prod"
          + "Role"        = "coordinator"
          + "Service"     = "citus"
        }
      + tags_all                             = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-coordinator-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
          + "Role"        = "coordinator"
          + "Service"     = "citus"
        }
      + tenancy                              = (known after apply)
      + user_data                            = "2a9e41ea765dcf3b3046ee10d2f458c18f00e430"
      + user_data_base64                     = (known after apply)
      + user_data_replace_on_change          = false
      + vpc_security_group_ids               = (known after apply)

      + ebs_block_device {
          + delete_on_termination = false
          + device_name           = "/dev/sdf"
          + encrypted             = true
          + iops                  = 3000
          + kms_key_id            = (known after apply)
          + snapshot_id           = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-coordinator-prod-data"
            }
          + tags_all              = (known after apply)
          + throughput            = 125
          + volume_id             = (known after apply)
          + volume_size           = 500
          + volume_type           = "gp3"
        }

      + root_block_device {
          + delete_on_termination = false
          + device_name           = (known after apply)
          + encrypted             = true
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-coordinator-prod-root"
            }
          + tags_all              = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = 100
          + volume_type           = "gp3"
        }
    }

  # module.citus_cluster.aws_instance.workers[0] will be created
  + resource "aws_instance" "workers" {
      + ami                                  = "ami-0b2aae5f4283c0df2"
      + arn                                  = (known after apply)
      + associate_public_ip_address          = (known after apply)
      + availability_zone                    = (known after apply)
      + cpu_core_count                       = (known after apply)
      + cpu_threads_per_core                 = (known after apply)
      + disable_api_stop                     = (known after apply)
      + disable_api_termination              = (known after apply)
      + ebs_optimized                        = (known after apply)
      + enable_primary_ipv6                  = (known after apply)
      + get_password_data                    = false
      + host_id                              = (known after apply)
      + host_resource_group_arn              = (known after apply)
      + iam_instance_profile                 = (known after apply)
      + id                                   = (known after apply)
      + instance_initiated_shutdown_behavior = (known after apply)
      + instance_lifecycle                   = (known after apply)
      + instance_state                       = (known after apply)
      + instance_type                        = "r6i.2xlarge"
      + ipv6_address_count                   = (known after apply)
      + ipv6_addresses                       = (known after apply)
      + key_name                             = (known after apply)
      + monitoring                           = (known after apply)
      + outpost_arn                          = (known after apply)
      + password_data                        = (known after apply)
      + placement_group                      = (known after apply)
      + placement_partition_number           = (known after apply)
      + primary_network_interface_id         = (known after apply)
      + private_dns                          = (known after apply)
      + private_ip                           = (known after apply)
      + public_dns                           = (known after apply)
      + public_ip                            = (known after apply)
      + secondary_private_ips                = (known after apply)
      + security_groups                      = (known after apply)
      + source_dest_check                    = true
      + spot_instance_request_id             = (known after apply)
      + subnet_id                            = "subnet-0a1f5a9a74ed030cf"
      + tags                                 = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-worker-0-prod"
          + "Role"        = "worker"
          + "Service"     = "citus"
          + "WorkerIndex" = "0"
        }
      + tags_all                             = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-worker-0-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
          + "Role"        = "worker"
          + "Service"     = "citus"
          + "WorkerIndex" = "0"
        }
      + tenancy                              = (known after apply)
      + user_data                            = "7b4bd87c9982aab7fa463c8d12e99399661f8bde"
      + user_data_base64                     = (known after apply)
      + user_data_replace_on_change          = false
      + vpc_security_group_ids               = (known after apply)

      + ebs_block_device {
          + delete_on_termination = false
          + device_name           = "/dev/sdf"
          + encrypted             = true
          + iops                  = 3000
          + kms_key_id            = (known after apply)
          + snapshot_id           = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-worker-0-prod-data"
            }
          + tags_all              = (known after apply)
          + throughput            = 125
          + volume_id             = (known after apply)
          + volume_size           = 500
          + volume_type           = "gp3"
        }

      + root_block_device {
          + delete_on_termination = false
          + device_name           = (known after apply)
          + encrypted             = true
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-worker-0-prod-root"
            }
          + tags_all              = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = 100
          + volume_type           = "gp3"
        }
    }

  # module.citus_cluster.aws_instance.workers[1] will be created
  + resource "aws_instance" "workers" {
      + ami                                  = "ami-0b2aae5f4283c0df2"
      + arn                                  = (known after apply)
      + associate_public_ip_address          = (known after apply)
      + availability_zone                    = (known after apply)
      + cpu_core_count                       = (known after apply)
      + cpu_threads_per_core                 = (known after apply)
      + disable_api_stop                     = (known after apply)
      + disable_api_termination              = (known after apply)
      + ebs_optimized                        = (known after apply)
      + enable_primary_ipv6                  = (known after apply)
      + get_password_data                    = false
      + host_id                              = (known after apply)
      + host_resource_group_arn              = (known after apply)
      + iam_instance_profile                 = (known after apply)
      + id                                   = (known after apply)
      + instance_initiated_shutdown_behavior = (known after apply)
      + instance_lifecycle                   = (known after apply)
      + instance_state                       = (known after apply)
      + instance_type                        = "r6i.2xlarge"
      + ipv6_address_count                   = (known after apply)
      + ipv6_addresses                       = (known after apply)
      + key_name                             = (known after apply)
      + monitoring                           = (known after apply)
      + outpost_arn                          = (known after apply)
      + password_data                        = (known after apply)
      + placement_group                      = (known after apply)
      + placement_partition_number           = (known after apply)
      + primary_network_interface_id         = (known after apply)
      + private_dns                          = (known after apply)
      + private_ip                           = (known after apply)
      + public_dns                           = (known after apply)
      + public_ip                            = (known after apply)
      + secondary_private_ips                = (known after apply)
      + security_groups                      = (known after apply)
      + source_dest_check                    = true
      + spot_instance_request_id             = (known after apply)
      + subnet_id                            = "subnet-02c930351cde1e9c3"
      + tags                                 = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-worker-1-prod"
          + "Role"        = "worker"
          + "Service"     = "citus"
          + "WorkerIndex" = "1"
        }
      + tags_all                             = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-worker-1-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
          + "Role"        = "worker"
          + "Service"     = "citus"
          + "WorkerIndex" = "1"
        }
      + tenancy                              = (known after apply)
      + user_data                            = "7b4bd87c9982aab7fa463c8d12e99399661f8bde"
      + user_data_base64                     = (known after apply)
      + user_data_replace_on_change          = false
      + vpc_security_group_ids               = (known after apply)

      + ebs_block_device {
          + delete_on_termination = false
          + device_name           = "/dev/sdf"
          + encrypted             = true
          + iops                  = 3000
          + kms_key_id            = (known after apply)
          + snapshot_id           = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-worker-1-prod-data"
            }
          + tags_all              = (known after apply)
          + throughput            = 125
          + volume_id             = (known after apply)
          + volume_size           = 500
          + volume_type           = "gp3"
        }

      + root_block_device {
          + delete_on_termination = false
          + device_name           = (known after apply)
          + encrypted             = true
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-worker-1-prod-root"
            }
          + tags_all              = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = 100
          + volume_type           = "gp3"
        }
    }

  # module.citus_cluster.aws_security_group.citus will be created
  + resource "aws_security_group" "citus" {
      + arn                    = (known after apply)
      + description            = "Security group for Citus cluster"
      + egress                 = [
          + {
              + cidr_blocks      = [
                  + "0.0.0.0/0",
                ]
              + description      = "Allow all outbound"
              + from_port        = 0
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "-1"
              + security_groups  = []
              + self             = false
              + to_port          = 0
            },
        ]
      + id                     = (known after apply)
      + ingress                = [
          + {
              + cidr_blocks      = [
                  + "10.0.0.0/8",
                ]
              + description      = "PgBouncer access"
              + from_port        = 6432
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 6432
            },
          + {
              + cidr_blocks      = [
                  + "10.0.0.0/8",
                ]
              + description      = "PostgreSQL access"
              + from_port        = 5432
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 5432
            },
          + {
              + cidr_blocks      = [
                  + "10.0.0.0/8",
                ]
              + description      = "SSH access"
              + from_port        = 22
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 22
            },
          + {
              + cidr_blocks      = []
              + description      = "Internal cluster communication"
              + from_port        = 0
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = true
              + to_port          = 65535
            },
        ]
      + name                   = (known after apply)
      + name_prefix            = "citus-prod-"
      + owner_id               = (known after apply)
      + revoke_rules_on_delete = false
      + tags                   = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-prod"
          + "Service"     = "citus"
        }
      + tags_all               = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
          + "Service"     = "citus"
        }
      + vpc_id                 = "vpc-0af7da1e7d94d62bd"
    }

Plan: 9 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + connection_string = (sensitive value)
  + coordinator_ip    = (known after apply)
  + worker_ips        = [
      + (known after apply),
      + (known after apply),
    ]

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"
[2026-02-05_09:27:48] βœ“ Terraform plan succeeded
[2026-02-05_09:27:48] Generating plan JSON...
[2026-02-05_09:27:51] βœ“ Terraform provisioning step completed successfully

Next step: Review the plan and apply with 'terraform apply tfplan'

3
01-prepare-environment local
βœ… SUCCEEDED
⏰ Started: 2026-02-05 09:27:51
🏁 Finished: 2026-02-05 09:28:09
⏱️ Duration: 18 seconds
πŸ“‹ Sub-steps (1): 0% complete
❓ 99-create-topology-from-form
πŸ“„ View Logs (6291 chars)
[INFO] FastOrder Environment Preparation
[INFO] Service: zone
[INFO] Zone: universe
[INFO] Environment: dev
[INFO] Branch: main
[INFO] State Directory: /opt/fastorder/bash/scripts/env_app_setup/state
[INFO] Library: /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] IP: 142.93.238.16 (specified)

[INFO] Creating environment using fo-env...
[INFO] Creating new FastOrder environment (v1 topology)
[INFO] Generated environment ID: zone-universe-main-dev
[INFO] Using provided IP: 142.93.238.16
[INFO] Allocated interface: eth0:16
[INFO] Configuring network interface for VM IP: 142.93.238.16
[INFO] VM IP 142.93.238.16 is already configured on eth0:16
[CONFIG] No web configuration found for environment: zone-universe-main-dev
[CONFIG] Using defaults: ES_NODES=1, PG_WORKERS=1
[INFO] Service enabled flags: db=yes, eventbus=yes, search=yes
[ OK ] Created topology.json at /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
[ OK ] Generated overlay configurations in /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/generated/
[ OK ] Updated environments.json
[ OK ] Updated setup.json
[ OK ] Environment created successfully!
[INFO] 
[INFO] Environment Details:
[INFO]   ID: zone-universe-main-dev
[INFO]   Service: zone
[INFO]   zone: universe
[INFO]   Environment: dev
[INFO]   Branch: main
[INFO]   IP: 142.93.238.16
[INFO]   Interface: eth0:16
[INFO] 
[INFO] Configuration files:
[INFO]   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
[INFO]   Generated: /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/generated/*.env
[INFO]   Overrides: /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/overrides/*.env
[INFO] 
[INFO] To use this environment:
[INFO]   export ENV_ID="zone-universe-main-dev"
[INFO]   source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO]   init_environment
[ OK ] Environment preparation completed successfully!
[INFO] Creating topology from web form submission...
[INFO] Using environment from web interface: zone-universe-main-dev
[2026-02-05 09:27:53] Using web-provided environment: zone-universe-main-dev
[2026-02-05 09:27:53] Service: zone, Zone: universe, Branch: main, Env: dev
[ OK ] Environment initialized successfully (mode: general)
[INFO] Creating topology.json from web form submission...
[INFO] DEBUG: Service enabled flags...
[INFO]   DB_ENABLED=yes
[INFO]   EVENTBUS_ENABLED=yes
[INFO]   SEARCH_ENABLED=yes
[INFO] DEBUG: Checking for form submission variables...
[INFO]   service_es_ip=10.100.1.4
[INFO]   service_es_fqdn=search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com
[INFO]   service_pg_coordinator_ip=10.100.1.14
[WARN] IP 10.100.1.4 is already allocated, allocating new IP for search
[INFO] Adding search: search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.249) [reallocated from 10.100.1.4]
[WARN] IP 10.100.1.6 is already allocated, allocating new IP for search-node-01
[INFO] Adding search-node-01: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.250) [reallocated from 10.100.1.6]
[WARN] IP 10.100.1.8 is already allocated, allocating new IP for eventbus-broker-01
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Adding eventbus-broker-01: eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com (10.100.1.71) [reallocated from 10.100.1.8]
[WARN] IP 10.100.1.10 is already allocated, allocating new IP for eventbus-connect
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Adding eventbus-connect: eventbus-zone-universe-main-dev-kafka-connect.fastorder.com (10.100.1.120) [reallocated from 10.100.1.10]
[WARN] IP 10.100.1.12 is already allocated, allocating new IP for schema-registry
[ERROR] No available IPs in range 10.100.1.50-250
[WARN] Skipping schema-registry - could not allocate IP
[WARN] IP 10.100.1.14 is already allocated, allocating new IP for pg-coordinator
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Adding pg-coordinator: db-zone-universe-main-dev-postgresql-coordinator.fastorder.com (10.100.1.114) [reallocated from 10.100.1.14]
[WARN] IP 10.100.1.16 is already allocated, allocating new IP for pgbouncer
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Adding pgbouncer: db-zone-universe-main-dev-postgresql-bouncer.fastorder.com (10.100.1.78) [reallocated from 10.100.1.16]
[WARN] IP 10.100.1.18 is already allocated, allocating new IP for obs
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Adding obs: obs-zone-universe-main-dev.fastorder.com (10.100.1.73) [reallocated from 10.100.1.18]
[ OK ] Topology created from form data
[INFO] Applications registered:
  βœ“ eventbus-broker-01: eventbus-zone-universe-main-dev-kafka-broker-01.fastorder.com (10.100.1.71)
  βœ“ eventbus-connect: eventbus-zone-universe-main-dev-kafka-connect.fastorder.com (10.100.1.120)
  βœ“ obs: obs-zone-universe-main-dev.fastorder.com (10.100.1.73)
  βœ“ pg-coordinator: db-zone-universe-main-dev-postgresql-coordinator.fastorder.com (10.100.1.114)
  βœ“ pgbouncer: db-zone-universe-main-dev-postgresql-bouncer.fastorder.com (10.100.1.78)
  βœ“ search: search-zone-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.249)
  βœ“ search-node-01: search-zone-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.250)
[ OK ] Topology created from form data

[INFO] Next steps:
[INFO] 1. Review the generated topology.json and configurations
[INFO] 2. Customize overrides/*.env files if needed
[INFO] 3. Run subsequent installation steps (02-install-postgresql, etc.)

[INFO] To use this environment in other scripts:
[INFO]   export ENV_ID="$(fo-env list | tail -n1 | awk '{print $1}')"
[INFO]   source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO]   init_environment
4
02-iam local
⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

πŸ“„ View Logs (0 chars)
Loading logs...
5
02-observability-cell local
❌ FAILED
⏰ Started: 2026-02-05 09:28:09
🏁 Finished: 2026-02-05 09:28:12
⏱️ Duration: 3 seconds
πŸ“‹ Sub-steps (4): 0% complete
❓ steps/01-create-secrets
❓ steps/02-generate-mtls-certs
❓ steps/09-configure-firewall
❓ steps/10-apply-audit-schema
πŸ“„ View Logs (3181 chars)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] πŸš€ OBSERVABILITY CELL PROVISIONING STARTED
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Script: 02-observability-cell/run.sh
[INFO] Timestamp: 2026-02-05 09:28:09 UTC
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Ensuring correct permissions for observability deployment...
[2026-02-05 09:28:09 UTC] USER=www-data EUID=0 PID=543332 ACTION=fsop ARGS=chmod 775 /var/log/fastorder
[2026-02-05 09:28:09 UTC] USER=www-data EUID=0 PID=543341 ACTION=fsop ARGS=chown www-data:www-data /var/log/fastorder
[2026-02-05 09:28:09 UTC] USER=www-data EUID=0 PID=543350 ACTION=fsop ARGS=touch /var/log/fastorder/provisioning-elevated.log
[2026-02-05 09:28:09 UTC] USER=www-data EUID=0 PID=543368 ACTION=fsop ARGS=chown www-data:www-data /var/log/fastorder/provisioning-elevated.log
[OK]   Log directory: /var/log/fastorder (775)
[OK]   Log file: provisioning-elevated.log (666)
[2026-02-05 09:28:09 UTC] USER=www-data EUID=0 PID=543377 ACTION=fsop ARGS=chmod 775 /opt/fastorder/bash/scripts/env_app_setup/state
[OK]   State directory: 775
[2026-02-05 09:28:09 UTC] USER=www-data EUID=0 PID=543386 ACTION=fsop ARGS=mkdir -p /etc/fastorder/observability/certs
[2026-02-05 09:28:09 UTC] USER=www-data EUID=0 PID=543395 ACTION=fsop ARGS=chmod 750 /etc/fastorder/observability/certs
[OK]   Cert directory: /etc/fastorder/observability/certs (750 - secure)
[OK]   Lib scripts: executable (755)
[OK]   All deployment scripts: executable (755)
[OK]   All directories: accessible (755)
[OK]   βœ… All permissions verified and fixed
[CREDS] Using AWS credentials from: /var/www/.aws/credentials
[CREDS] Credential management library loaded (region: me-central-1)
[INFO] Using web-provided environment: zone-universe-main-dev
[INFO] Loaded environment: zone-universe-main-dev (svc=zone zone=universe env=dev ip=142.93.238.16)
═══════════════════════════════════════════════════════════════════════════════
OBSERVABILITY CELL PROVISIONING
═══════════════════════════════════════════════════════════════════════════════
[INFO] Application Cell: zone-universe-main-dev
[INFO] Observability Cell: obs-zone-universe-main-dev
[INFO] Service: zone | Zone: universe | Env: dev

[INFO] Step 1/10: Provisioning network infrastructure...
[INFO]   Using existing IP for obs: 10.100.1.73
/opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/run.sh: line 250: echo: write error: Broken pipe
[INFO]   Allocated new IP for metrics: 10.100.1.53
[2026-02-05 09:28:11 UTC] USER=www-data EUID=0 PID=543608 ACTION=fsop ARGS=cp /tmp/tmp.17Z8iiBsK0 /opt/fastorder/bash/scripts/env_app_setup/state/zone-universe-main-dev/topology.json
/opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/run.sh: line 250: echo: write error: Broken pipe
[INFO]   Allocated new IP for dashboards: 10.100.1.194
[ERROR] Failed to allocate IP for logstore - no available IPs in range
[ERROR] Failed to allocate observability IPs
6
03-search local
⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

πŸ“„ View Logs (0 chars)
Loading logs...
7
04-eventbus local
⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

πŸ“„ View Logs (0 chars)
Loading logs...
8
05-db local
⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

πŸ“„ View Logs (0 chars)
Loading logs...
9
06-finalizing local
⏸️ PENDING

⏳ This step is pending and will execute after the previous steps complete successfully.

πŸ“„ View Logs (0 chars)
Loading logs...
9
Total Steps
3
Succeeded
1
Failed
0
Running
5
Pending
44 seconds
Total Steps Time
← Back to Dashboard πŸ” View Environment