📊 Provisioning Job Status

════════════════════════════════════════════════════════════════
  FastOrder Pre-Flight Validation Checks
════════════════════════════════════════════════════════════════

[INFO] Checking SSH connectivity to target host...
[✓] Target is localhost, skipping SSH check

[INFO] Checking available disk space...
[✓] Disk space sufficient: 58GB available (required: 50GB)

[INFO] Checking available memory...
[⚠] Memory limited: 15GB (recommended: 16GB)
  → Consider reducing Elasticsearch nodes or PostgreSQL workers

[INFO] Checking critical port availability...
[✓] Port 5432 in use on specific IP (10.100.1.190:5432) - OK, can use different IP
[✓] Port 9200 in use on specific IP ([::ffff:10.100.1.181]) - OK, can use different IP
[✓] Port 9300 in use on specific IP ([::ffff:10.100.1.181]) - OK, can use different IP
[✓] Port 9092 in use on specific IP ([::ffff:10.100.1.213]) - OK, can use different IP
[✓] Port 2181 available (Zookeeper)

[INFO] Checking DNS resolution...
[✓] DNS resolution working: google.com
[✓] DNS resolution working: github.com
[✓] DNS resolution working: archive.ubuntu.com

[INFO] Checking required system commands...
[✓] Command available: curl
[✓] Command available: wget
[✓] Command available: git
[✓] Command available: sudo
[✓] Command available: systemctl
[✓] Command available: apt-get

[INFO] Checking current system load...
[⚠] System load elevated: 3.27 (4 CPUs)
  → Provisioning may be slower than expected

[INFO] Checking for existing environment conflicts...
[✓] No conflicting services found for: user-uae-main-dev

════════════════════════════════════════════════════════════════
  Pre-Flight Check Summary
════════════════════════════════════════════════════════════════
[⚠] 2 warning(s) detected

⚠️  Environment can proceed with caution
   Review warnings above and consider remediation

[INFO] Using web-provided environment: user-universe-main-dev
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
[2026-01-12_20:35:25] Starting Terraform provisioning step
[2026-01-12_20:35:25] Service: user
[2026-01-12_20:35:25] Zone: universe
[2026-01-12_20:35:25] Environment: dev
[2026-01-12_20:35:25] Resource: web-03
[2026-01-12_20:35:25] Terraform binary: /home/ab/bin/terraform
[2026-01-12_20:35:25] HOME: /home/www-data
[2026-01-12_20:35:25] AWS Config: /home/ab/.aws/config
[2026-01-12_20:35:25] AWS Credentials: /home/ab/.aws/credentials
[2026-01-12_20:35:25] Terraform directory: /opt/fastorder/cli/terraform/examples/citus-production
[2026-01-12_20:35:25] Running terraform init...

Initializing the backend...
Upgrading modules...
- citus_cluster in ../../modules/citus_cluster

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 5.0"...
- Using previously-installed hashicorp/aws v5.100.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
[2026-01-12_20:35:30] ✓ Terraform init succeeded
[2026-01-12_20:35:30] Running terraform validate...
Success! The configuration is valid.

[2026-01-12_20:35:34] ✓ Terraform validate succeeded
[2026-01-12_20:35:34] Running terraform plan...
module.citus_cluster.data.aws_caller_identity.current: Reading...
module.citus_cluster.data.aws_caller_identity.current: Read complete after 0s [id=464621692046]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.citus_cluster.aws_iam_instance_profile.citus will be created
  + resource "aws_iam_instance_profile" "citus" {
      + arn         = (known after apply)
      + create_date = (known after apply)
      + id          = (known after apply)
      + name        = (known after apply)
      + name_prefix = "citus-prod-"
      + path        = "/"
      + role        = (known after apply)
      + tags        = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-prod"
        }
      + tags_all    = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
        }
      + unique_id   = (known after apply)
    }

  # module.citus_cluster.aws_iam_role.citus will be created
  + resource "aws_iam_role" "citus" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRole"
                      + Effect    = "Allow"
                      + Principal = {
                          + Service = "ec2.amazonaws.com"
                        }
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = (known after apply)
      + name_prefix           = "citus-prod-"
      + path                  = "/"
      + tags                  = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-prod"
        }
      + tags_all              = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
        }
      + unique_id             = (known after apply)
    }

  # module.citus_cluster.aws_iam_role_policy.secrets_manager[0] will be created
  + resource "aws_iam_role_policy" "secrets_manager" {
      + id          = (known after apply)
      + name        = (known after apply)
      + name_prefix = "secrets-access-"
      + policy      = jsonencode(
            {
              + Statement = [
                  + {
                      + Action   = [
                          + "secretsmanager:GetSecretValue",
                          + "secretsmanager:DescribeSecret",
                        ]
                      + Effect   = "Allow"
                      + Resource = "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/db/web/ksa/main/dev/postgresqladmin/ksa/prod*"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + role        = (known after apply)
    }

  # module.citus_cluster.aws_iam_role_policy_attachment.cloudwatch will be created
  + resource "aws_iam_role_policy_attachment" "cloudwatch" {
      + id         = (known after apply)
      + policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
      + role       = (known after apply)
    }

  # module.citus_cluster.aws_iam_role_policy_attachment.ssm will be created
  + resource "aws_iam_role_policy_attachment" "ssm" {
      + id         = (known after apply)
      + policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
      + role       = (known after apply)
    }

  # module.citus_cluster.aws_instance.coordinator will be created
  + resource "aws_instance" "coordinator" {
      + ami                                  = "ami-0b2aae5f4283c0df2"
      + arn                                  = (known after apply)
      + associate_public_ip_address          = (known after apply)
      + availability_zone                    = (known after apply)
      + cpu_core_count                       = (known after apply)
      + cpu_threads_per_core                 = (known after apply)
      + disable_api_stop                     = (known after apply)
      + disable_api_termination              = (known after apply)
      + ebs_optimized                        = (known after apply)
      + enable_primary_ipv6                  = (known after apply)
      + get_password_data                    = false
      + host_id                              = (known after apply)
      + host_resource_group_arn              = (known after apply)
      + iam_instance_profile                 = (known after apply)
      + id                                   = (known after apply)
      + instance_initiated_shutdown_behavior = (known after apply)
      + instance_lifecycle                   = (known after apply)
      + instance_state                       = (known after apply)
      + instance_type                        = "r6i.2xlarge"
      + ipv6_address_count                   = (known after apply)
      + ipv6_addresses                       = (known after apply)
      + key_name                             = (known after apply)
      + monitoring                           = (known after apply)
      + outpost_arn                          = (known after apply)
      + password_data                        = (known after apply)
      + placement_group                      = (known after apply)
      + placement_partition_number           = (known after apply)
      + primary_network_interface_id         = (known after apply)
      + private_dns                          = (known after apply)
      + private_ip                           = (known after apply)
      + public_dns                           = (known after apply)
      + public_ip                            = (known after apply)
      + secondary_private_ips                = (known after apply)
      + security_groups                      = (known after apply)
      + source_dest_check                    = true
      + spot_instance_request_id             = (known after apply)
      + subnet_id                            = "subnet-0a1f5a9a74ed030cf"
      + tags                                 = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-coordinator-prod"
          + "Role"        = "coordinator"
          + "Service"     = "citus"
        }
      + tags_all                             = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-coordinator-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
          + "Role"        = "coordinator"
          + "Service"     = "citus"
        }
      + tenancy                              = (known after apply)
      + user_data                            = "2a9e41ea765dcf3b3046ee10d2f458c18f00e430"
      + user_data_base64                     = (known after apply)
      + user_data_replace_on_change          = false
      + vpc_security_group_ids               = (known after apply)

      + ebs_block_device {
          + delete_on_termination = false
          + device_name           = "/dev/sdf"
          + encrypted             = true
          + iops                  = 3000
          + kms_key_id            = (known after apply)
          + snapshot_id           = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-coordinator-prod-data"
            }
          + tags_all              = (known after apply)
          + throughput            = 125
          + volume_id             = (known after apply)
          + volume_size           = 500
          + volume_type           = "gp3"
        }

      + root_block_device {
          + delete_on_termination = false
          + device_name           = (known after apply)
          + encrypted             = true
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-coordinator-prod-root"
            }
          + tags_all              = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = 100
          + volume_type           = "gp3"
        }
    }

  # module.citus_cluster.aws_instance.workers[0] will be created
  + resource "aws_instance" "workers" {
      + ami                                  = "ami-0b2aae5f4283c0df2"
      + arn                                  = (known after apply)
      + associate_public_ip_address          = (known after apply)
      + availability_zone                    = (known after apply)
      + cpu_core_count                       = (known after apply)
      + cpu_threads_per_core                 = (known after apply)
      + disable_api_stop                     = (known after apply)
      + disable_api_termination              = (known after apply)
      + ebs_optimized                        = (known after apply)
      + enable_primary_ipv6                  = (known after apply)
      + get_password_data                    = false
      + host_id                              = (known after apply)
      + host_resource_group_arn              = (known after apply)
      + iam_instance_profile                 = (known after apply)
      + id                                   = (known after apply)
      + instance_initiated_shutdown_behavior = (known after apply)
      + instance_lifecycle                   = (known after apply)
      + instance_state                       = (known after apply)
      + instance_type                        = "r6i.2xlarge"
      + ipv6_address_count                   = (known after apply)
      + ipv6_addresses                       = (known after apply)
      + key_name                             = (known after apply)
      + monitoring                           = (known after apply)
      + outpost_arn                          = (known after apply)
      + password_data                        = (known after apply)
      + placement_group                      = (known after apply)
      + placement_partition_number           = (known after apply)
      + primary_network_interface_id         = (known after apply)
      + private_dns                          = (known after apply)
      + private_ip                           = (known after apply)
      + public_dns                           = (known after apply)
      + public_ip                            = (known after apply)
      + secondary_private_ips                = (known after apply)
      + security_groups                      = (known after apply)
      + source_dest_check                    = true
      + spot_instance_request_id             = (known after apply)
      + subnet_id                            = "subnet-0a1f5a9a74ed030cf"
      + tags                                 = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-worker-0-prod"
          + "Role"        = "worker"
          + "Service"     = "citus"
          + "WorkerIndex" = "0"
        }
      + tags_all                             = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-worker-0-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
          + "Role"        = "worker"
          + "Service"     = "citus"
          + "WorkerIndex" = "0"
        }
      + tenancy                              = (known after apply)
      + user_data                            = "7b4bd87c9982aab7fa463c8d12e99399661f8bde"
      + user_data_base64                     = (known after apply)
      + user_data_replace_on_change          = false
      + vpc_security_group_ids               = (known after apply)

      + ebs_block_device {
          + delete_on_termination = false
          + device_name           = "/dev/sdf"
          + encrypted             = true
          + iops                  = 3000
          + kms_key_id            = (known after apply)
          + snapshot_id           = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-worker-0-prod-data"
            }
          + tags_all              = (known after apply)
          + throughput            = 125
          + volume_id             = (known after apply)
          + volume_size           = 500
          + volume_type           = "gp3"
        }

      + root_block_device {
          + delete_on_termination = false
          + device_name           = (known after apply)
          + encrypted             = true
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-worker-0-prod-root"
            }
          + tags_all              = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = 100
          + volume_type           = "gp3"
        }
    }

  # module.citus_cluster.aws_instance.workers[1] will be created
  + resource "aws_instance" "workers" {
      + ami                                  = "ami-0b2aae5f4283c0df2"
      + arn                                  = (known after apply)
      + associate_public_ip_address          = (known after apply)
      + availability_zone                    = (known after apply)
      + cpu_core_count                       = (known after apply)
      + cpu_threads_per_core                 = (known after apply)
      + disable_api_stop                     = (known after apply)
      + disable_api_termination              = (known after apply)
      + ebs_optimized                        = (known after apply)
      + enable_primary_ipv6                  = (known after apply)
      + get_password_data                    = false
      + host_id                              = (known after apply)
      + host_resource_group_arn              = (known after apply)
      + iam_instance_profile                 = (known after apply)
      + id                                   = (known after apply)
      + instance_initiated_shutdown_behavior = (known after apply)
      + instance_lifecycle                   = (known after apply)
      + instance_state                       = (known after apply)
      + instance_type                        = "r6i.2xlarge"
      + ipv6_address_count                   = (known after apply)
      + ipv6_addresses                       = (known after apply)
      + key_name                             = (known after apply)
      + monitoring                           = (known after apply)
      + outpost_arn                          = (known after apply)
      + password_data                        = (known after apply)
      + placement_group                      = (known after apply)
      + placement_partition_number           = (known after apply)
      + primary_network_interface_id         = (known after apply)
      + private_dns                          = (known after apply)
      + private_ip                           = (known after apply)
      + public_dns                           = (known after apply)
      + public_ip                            = (known after apply)
      + secondary_private_ips                = (known after apply)
      + security_groups                      = (known after apply)
      + source_dest_check                    = true
      + spot_instance_request_id             = (known after apply)
      + subnet_id                            = "subnet-02c930351cde1e9c3"
      + tags                                 = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-worker-1-prod"
          + "Role"        = "worker"
          + "Service"     = "citus"
          + "WorkerIndex" = "1"
        }
      + tags_all                             = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-worker-1-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
          + "Role"        = "worker"
          + "Service"     = "citus"
          + "WorkerIndex" = "1"
        }
      + tenancy                              = (known after apply)
      + user_data                            = "7b4bd87c9982aab7fa463c8d12e99399661f8bde"
      + user_data_base64                     = (known after apply)
      + user_data_replace_on_change          = false
      + vpc_security_group_ids               = (known after apply)

      + ebs_block_device {
          + delete_on_termination = false
          + device_name           = "/dev/sdf"
          + encrypted             = true
          + iops                  = 3000
          + kms_key_id            = (known after apply)
          + snapshot_id           = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-worker-1-prod-data"
            }
          + tags_all              = (known after apply)
          + throughput            = 125
          + volume_id             = (known after apply)
          + volume_size           = 500
          + volume_type           = "gp3"
        }

      + root_block_device {
          + delete_on_termination = false
          + device_name           = (known after apply)
          + encrypted             = true
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-worker-1-prod-root"
            }
          + tags_all              = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = 100
          + volume_type           = "gp3"
        }
    }

  # module.citus_cluster.aws_security_group.citus will be created
  + resource "aws_security_group" "citus" {
      + arn                    = (known after apply)
      + description            = "Security group for Citus cluster"
      + egress                 = [
          + {
              + cidr_blocks      = [
                  + "0.0.0.0/0",
                ]
              + description      = "Allow all outbound"
              + from_port        = 0
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "-1"
              + security_groups  = []
              + self             = false
              + to_port          = 0
            },
        ]
      + id                     = (known after apply)
      + ingress                = [
          + {
              + cidr_blocks      = [
                  + "10.0.0.0/8",
                ]
              + description      = "PgBouncer access"
              + from_port        = 6432
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 6432
            },
          + {
              + cidr_blocks      = [
                  + "10.0.0.0/8",
                ]
              + description      = "PostgreSQL access"
              + from_port        = 5432
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 5432
            },
          + {
              + cidr_blocks      = [
                  + "10.0.0.0/8",
                ]
              + description      = "SSH access"
              + from_port        = 22
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 22
            },
          + {
              + cidr_blocks      = []
              + description      = "Internal cluster communication"
              + from_port        = 0
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = true
              + to_port          = 65535
            },
        ]
      + name                   = (known after apply)
      + name_prefix            = "citus-prod-"
      + owner_id               = (known after apply)
      + revoke_rules_on_delete = false
      + tags                   = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-prod"
          + "Service"     = "citus"
        }
      + tags_all               = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
          + "Service"     = "citus"
        }
      + vpc_id                 = "vpc-0af7da1e7d94d62bd"
    }

Plan: 9 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + connection_string = (sensitive value)
  + coordinator_ip    = (known after apply)
  + worker_ips        = [
      + (known after apply),
      + (known after apply),
    ]

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"
[2026-01-12_20:35:40] ✓ Terraform plan succeeded
[2026-01-12_20:35:40] Generating plan JSON...
[2026-01-12_20:35:43] ✓ Terraform provisioning step completed successfully

Next step: Review the plan and apply with 'terraform apply tfplan'

[INFO] FastOrder Environment Preparation
[INFO] Service: user
[INFO] Zone: universe
[INFO] Environment: dev
[INFO] Branch: main
[INFO] State Directory: /opt/fastorder/bash/scripts/env_app_setup/state
[INFO] Library: /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] IP: 142.93.238.16 (specified)

[INFO] Creating environment using fo-env...
[INFO] Creating new FastOrder environment (v1 topology)
[INFO] Generated environment ID: user-universe-main-dev
[INFO] Using provided IP: 142.93.238.16
[INFO] Allocated interface: eth0:16
[INFO] Configuring network interface for VM IP: 142.93.238.16
[INFO] VM IP 142.93.238.16 is already configured on eth0:16
[CONFIG] No web configuration found for environment: user-universe-main-dev
[CONFIG] Using defaults: ES_NODES=1, PG_WORKERS=1
[INFO] Service enabled flags: db=no, eventbus=no, search=yes
[ OK ] Created topology.json at /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
[ OK ] Generated overlay configurations in /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/generated/
[ OK ] Updated environments.json
[ OK ] Updated setup.json
[ OK ] Environment created successfully!
[INFO] 
[INFO] Environment Details:
[INFO]   ID: user-universe-main-dev
[INFO]   Service: user
[INFO]   zone: universe
[INFO]   Environment: dev
[INFO]   Branch: main
[INFO]   IP: 142.93.238.16
[INFO]   Interface: eth0:16
[INFO] 
[INFO] Configuration files:
[INFO]   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
[INFO]   Generated: /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/generated/*.env
[INFO]   Overrides: /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/overrides/*.env
[INFO] 
[INFO] To use this environment:
[INFO]   export ENV_ID="user-universe-main-dev"
[INFO]   source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO]   init_environment
[ OK ] Environment preparation completed successfully!
[INFO] Creating topology from web form submission...
[INFO] Loaded from topology.json: user-universe-main-dev
[2026-01-12 20:35:45] Loaded environment: user-universe-main-dev
[2026-01-12 20:35:45] Service: user, Zone: universe, Branch: main, Env: dev
[2026-01-12 20:35:45] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-12 20:35:45] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-12 20:35:45] PostgreSQL HA Nodes: 1, Citus Enabled: yes
[ OK ] Environment initialized successfully (mode: general)
[INFO] Creating topology.json from web form submission...
[INFO] DEBUG: Service enabled flags...
[INFO]   DB_ENABLED=no
[INFO]   EVENTBUS_ENABLED=no
[INFO]   SEARCH_ENABLED=yes
[INFO] DEBUG: Checking for form submission variables...
[INFO]   service_es_ip=10.100.1.4
[INFO]   service_es_fqdn=search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com
[INFO]   service_pg_coordinator_ip=NOT SET
[WARN] IP 10.100.1.4 is already allocated, allocating new IP for search
[INFO] Adding search: search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.218) [reallocated from 10.100.1.4]
[WARN] IP 10.100.1.6 is already allocated, allocating new IP for search-node-01
[INFO] Adding search-node-01: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.219) [reallocated from 10.100.1.6]
[INFO] Skipping Kafka/EventBus - EVENTBUS_ENABLED=no
[INFO] Skipping PostgreSQL/PgBouncer - DB_ENABLED=no
[WARN] IP 10.100.1.8 is already allocated, allocating new IP for obs
[INFO] Adding obs: obs-user-universe-main-dev.fastorder.com (10.100.1.220) [reallocated from 10.100.1.8]
[ OK ] Topology created from form data
[INFO] Applications registered:
  ✓ obs: obs-user-universe-main-dev.fastorder.com (10.100.1.220)
  ✓ search: search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.218)
  ✓ search-node-01: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.219)
[ OK ] Topology created from form data

[INFO] Next steps:
[INFO] 1. Review the generated topology.json and configurations
[INFO] 2. Customize overrides/*.env files if needed
[INFO] 3. Run subsequent installation steps (02-install-postgresql, etc.)

[INFO] To use this environment in other scripts:
[INFO]   export ENV_ID="$(fo-env list | tail -n1 | awk '{print $1}')"
[INFO]   source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO]   init_environment

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🚀 OBSERVABILITY CELL PROVISIONING STARTED
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Script: 02-observability-cell/run.sh
[INFO] Timestamp: 2026-01-12 20:35:48 UTC
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Ensuring correct permissions for observability deployment...
[2026-01-12 20:35:48 UTC] USER=www-data EUID=0 PID=683770 ACTION=fsop ARGS=chmod 775 /var/log/fastorder
[2026-01-12 20:35:48 UTC] USER=www-data EUID=0 PID=683785 ACTION=fsop ARGS=chown www-data:www-data /var/log/fastorder
[2026-01-12 20:35:48 UTC] USER=www-data EUID=0 PID=683798 ACTION=fsop ARGS=touch /var/log/fastorder/provisioning-elevated.log
[2026-01-12 20:35:48 UTC] USER=www-data EUID=0 PID=683807 ACTION=fsop ARGS=chmod 666 /var/log/fastorder/provisioning-elevated.log
[2026-01-12 20:35:48 UTC] USER=www-data EUID=0 PID=683816 ACTION=fsop ARGS=chown www-data:www-data /var/log/fastorder/provisioning-elevated.log
[OK]   Log directory: /var/log/fastorder (775)
[OK]   Log file: provisioning-elevated.log (666)
[2026-01-12 20:35:48 UTC] USER=www-data EUID=0 PID=683826 ACTION=fsop ARGS=chmod 775 /opt/fastorder/bash/scripts/env_app_setup/state
[OK]   State directory: 775
[2026-01-12 20:35:48 UTC] USER=www-data EUID=0 PID=683835 ACTION=fsop ARGS=mkdir -p /etc/fastorder/observability/certs
[2026-01-12 20:35:49 UTC] USER=www-data EUID=0 PID=683844 ACTION=fsop ARGS=chmod 750 /etc/fastorder/observability/certs
[OK]   Cert directory: /etc/fastorder/observability/certs (750 - secure)
[OK]   Lib scripts: executable (755)
[OK]   All deployment scripts: executable (755)
[OK]   All directories: accessible (755)
[OK]   ✅ All permissions verified and fixed
[CREDS] Using AWS credentials from: /var/www/.aws/credentials
[CREDS] Credential management library loaded (region: me-central-1)
[INFO] Using web-provided environment: user-universe-main-dev
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
═══════════════════════════════════════════════════════════════════════════════
OBSERVABILITY CELL PROVISIONING
═══════════════════════════════════════════════════════════════════════════════
[INFO] Application Cell: user-universe-main-dev
[INFO] Observability Cell: obs-user-universe-main-dev
[INFO] Service: user | Zone: universe | Env: dev

[INFO] Step 1/10: Provisioning network infrastructure...
[INFO]   Using existing IP for obs: 10.100.1.220
[INFO]   Allocated new IP for metrics: 10.100.1.221
[2026-01-12 20:35:50 UTC] USER=www-data EUID=0 PID=684354 ACTION=fsop ARGS=cp /tmp/tmp.973uHOxpct /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
[INFO]   Allocated new IP for dashboards: 10.100.1.222
[2026-01-12 20:35:50 UTC] USER=www-data EUID=0 PID=684372 ACTION=fsop ARGS=cp /tmp/tmp.U2625fsXAM /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
[INFO]   Allocated new IP for logstore: 10.100.1.223
[2026-01-12 20:35:50 UTC] USER=www-data EUID=0 PID=684395 ACTION=fsop ARGS=cp /tmp/tmp.zzkC4gLrgA /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
[INFO]   Allocated new IP for traces: 10.100.1.224
[2026-01-12 20:35:50 UTC] USER=www-data EUID=0 PID=684416 ACTION=fsop ARGS=cp /tmp/tmp.bKuvbGIYD8 /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
[INFO]   Allocated new IP for alerts: 10.100.1.225
[2026-01-12 20:35:50 UTC] USER=www-data EUID=0 PID=684434 ACTION=fsop ARGS=cp /tmp/tmp.2om8zVN63N /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
[INFO]   Allocated new IP for telemetry: 10.100.1.226
[2026-01-12 20:35:50 UTC] USER=www-data EUID=0 PID=684451 ACTION=fsop ARGS=cp /tmp/tmp.5VkClFOWzh /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
[INFO]   Allocated observability IPs:
[INFO]     metrics: 10.100.1.221
[INFO]     alerts: 10.100.1.225
[INFO]     dashboards: 10.100.1.222
[INFO]     traces: 10.100.1.224
[INFO]     telemetry: 10.100.1.226
[INFO]     logstore: 10.100.1.223
[INFO]     proxy: 10.100.1.220
[INFO]     obs: 10.100.1.220
[ OK ] Network infrastructure allocated
[INFO] Cleaning up ports from previous environments...
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-user-universe-main-dev
[INFO] IP Address: 10.100.1.220
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service clickhouse-server@obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server@obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service prometheus-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service grafana-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service tempo-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service alertmanager-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Found 24 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.220...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup completed successfully
[INFO] Configuring IP aliases on network interface...
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING NETWORK IP ALIASES
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Interface: lo
[INFO] IP Count: 8

[INFO] Configuring: metrics → 10.100.1.221
[INFO]   Configuring IP alias: 10.100.1.221/32 on lo
[OK]     ✅ IP 10.100.1.221 configured successfully on lo
[OK]     ✅ IP 10.100.1.221 verified on network interface
[INFO] Configuring: alerts → 10.100.1.225
[INFO]   Configuring IP alias: 10.100.1.225/32 on lo
[OK]     ✅ IP 10.100.1.225 configured successfully on lo
[OK]     ✅ IP 10.100.1.225 verified on network interface
[INFO] Configuring: dashboards → 10.100.1.222
[INFO]   Configuring IP alias: 10.100.1.222/32 on lo
[OK]     ✅ IP 10.100.1.222 configured successfully on lo
[OK]     ✅ IP 10.100.1.222 verified on network interface
[INFO] Configuring: traces → 10.100.1.224
[INFO]   Configuring IP alias: 10.100.1.224/32 on lo
[OK]     ✅ IP 10.100.1.224 configured successfully on lo
[OK]     ✅ IP 10.100.1.224 verified on network interface
[INFO] Configuring: telemetry → 10.100.1.226
[INFO]   Configuring IP alias: 10.100.1.226/32 on lo
[OK]     ✅ IP 10.100.1.226 configured successfully on lo
[OK]     ✅ IP 10.100.1.226 verified on network interface
[INFO] Configuring: logstore → 10.100.1.223
[INFO]   Configuring IP alias: 10.100.1.223/32 on lo
[OK]     ✅ IP 10.100.1.223 configured successfully on lo
[OK]     ✅ IP 10.100.1.223 verified on network interface
[INFO] Configuring: proxy → 10.100.1.220
[INFO]   Configuring IP alias: 10.100.1.220/32 on lo
[OK]     ✅ IP 10.100.1.220 configured successfully on lo
[OK]     ✅ IP 10.100.1.220 verified on network interface
[INFO] Configuring: obs → 10.100.1.220
[INFO]   IP 10.100.1.220 already configured on network interface

[OK]   ═══════════════════════════════════════════════════════════════
[OK]   ✅ All IP aliases configured successfully
[OK]   ═══════════════════════════════════════════════════════════════
[INFO] Current IP configuration on lo:
      inet 127.0.0.1/8 scope host lo
      inet 10.100.1.204/32 scope global lo:pgbouncer
      inet 10.100.1.192/32 scope global lo:pgbouncer
      inet 10.100.60.2/32 scope global lo
      inet 10.100.1.155/32 scope global lo
      inet 10.100.1.183/32 scope global lo
      inet 10.100.1.187/32 scope global lo
      inet 10.100.1.217/32 scope global lo
      inet 10.100.1.184/32 scope global lo
      inet 10.100.1.111/32 scope global lo
      inet 10.100.1.181/32 scope global lo
      inet 10.100.1.221/32 scope global lo
      inet 10.100.1.225/32 scope global lo
      inet 10.100.1.222/32 scope global lo
      inet 10.100.1.224/32 scope global lo
      inet 10.100.1.226/32 scope global lo
      inet 10.100.1.223/32 scope global lo
      inet 10.100.1.220/32 scope global lo

[OK]   IP aliases configured on network interface
[INFO] Step 2/10: Creating DNS entries...
[INFO] Configuring DNS entries in /etc/hosts...
[2026-01-12 20:35:51 UTC] USER=www-data EUID=0 PID=684813 ACTION=fsop ARGS=sed -i /metrics-user-universe-main-dev-prometheus.fastorder.com/d /etc/hosts
[INFO]   Added: metrics-user-universe-main-dev-prometheus.fastorder.com → 10.100.1.221
[2026-01-12 20:35:51 UTC] USER=www-data EUID=0 PID=684833 ACTION=fsop ARGS=sed -i /alerts-user-universe-main-dev-alertmanager.fastorder.com/d /etc/hosts
[INFO]   Added: alerts-user-universe-main-dev-alertmanager.fastorder.com → 10.100.1.225
[2026-01-12 20:35:51 UTC] USER=www-data EUID=0 PID=684853 ACTION=fsop ARGS=sed -i /dashboards-user-universe-main-dev-grafana.fastorder.com/d /etc/hosts
[INFO]   Added: dashboards-user-universe-main-dev-grafana.fastorder.com → 10.100.1.222
[2026-01-12 20:35:51 UTC] USER=www-data EUID=0 PID=684873 ACTION=fsop ARGS=sed -i /traces-user-universe-main-dev-tempo.fastorder.com/d /etc/hosts
[INFO]   Added: traces-user-universe-main-dev-tempo.fastorder.com → 10.100.1.224
[2026-01-12 20:35:51 UTC] USER=www-data EUID=0 PID=684893 ACTION=fsop ARGS=sed -i /telemetry-user-universe-main-dev-opentelemetry.fastorder.com/d /etc/hosts
[INFO]   Added: telemetry-user-universe-main-dev-opentelemetry.fastorder.com → 10.100.1.226
[2026-01-12 20:35:51 UTC] USER=www-data EUID=0 PID=684913 ACTION=fsop ARGS=sed -i /logstore-user-universe-main-dev-clickhouse.fastorder.com/d /etc/hosts
[INFO]   Added: logstore-user-universe-main-dev-clickhouse.fastorder.com → 10.100.1.223
[2026-01-12 20:35:51 UTC] USER=www-data EUID=0 PID=684933 ACTION=fsop ARGS=sed -i /observe-user-universe-main-dev.fastorder.com/d /etc/hosts
[INFO]   Added: observe-user-universe-main-dev.fastorder.com → 10.100.1.220
[INFO] Adding observability integration aliases...
[2026-01-12 20:35:51 UTC] USER=www-data EUID=0 PID=684953 ACTION=fsop ARGS=sed -i /metrics-user-universe-main-dev.fastorder.com/d /etc/hosts
[INFO]   Added alias: metrics-user-universe-main-dev.fastorder.com → 10.100.1.221
[2026-01-12 20:35:51 UTC] USER=www-data EUID=0 PID=684973 ACTION=fsop ARGS=sed -i /alerts-user-universe-main-dev.fastorder.com/d /etc/hosts
[INFO]   Added alias: alerts-user-universe-main-dev.fastorder.com → 10.100.1.225
[2026-01-12 20:35:51 UTC] USER=www-data EUID=0 PID=684993 ACTION=fsop ARGS=sed -i /dashboards-user-universe-main-dev.fastorder.com/d /etc/hosts
[INFO]   Added alias: dashboards-user-universe-main-dev.fastorder.com → 10.100.1.222
[2026-01-12 20:35:51 UTC] USER=www-data EUID=0 PID=685013 ACTION=fsop ARGS=sed -i /traces-user-universe-main-dev.fastorder.com/d /etc/hosts
[INFO]   Added alias: traces-user-universe-main-dev.fastorder.com → 10.100.1.224
[2026-01-12 20:35:51 UTC] USER=www-data EUID=0 PID=685033 ACTION=fsop ARGS=sed -i /telemetry-user-universe-main-dev.fastorder.com/d /etc/hosts
[INFO]   Added alias: telemetry-user-universe-main-dev.fastorder.com → 10.100.1.226
[2026-01-12 20:35:51 UTC] USER=www-data EUID=0 PID=685055 ACTION=fsop ARGS=sed -i /logstore-user-universe-main-dev.fastorder.com/d /etc/hosts
[INFO]   Added alias: logstore-user-universe-main-dev.fastorder.com → 10.100.1.223
[2026-01-12 20:35:51 UTC] USER=www-data EUID=0 PID=685078 ACTION=fsop ARGS=sed -i /observe-user-universe-main-dev.fastorder.com/d /etc/hosts
[INFO]   Added alias: observe-user-universe-main-dev.fastorder.com → 10.100.1.220
[OK]   DNS entries created
[INFO] Step 3/10: Creating AWS Secrets Manager structure...
[INFO] Creating AWS Secrets Manager structure
[INFO]   Base path: fastorder/observability/user/universe/dev
[INFO]   Observability Cell: obs-user-universe-main-dev
[INFO]   Application Cell: user-universe-main-dev
[INFO]   Exists: fastorder/observability/user/universe/dev/metrics
[INFO]   Exists: fastorder/observability/user/universe/dev/dashboards
[INFO]   Exists: fastorder/observability/user/universe/dev/logstore
[INFO]   Exists: fastorder/observability/user/universe/dev/traces
[INFO]   Exists: fastorder/observability/user/universe/dev/telemetry
[INFO]   Exists: fastorder/observability/user/universe/dev/alerts
[INFO] Secrets structure created successfully
[OK]   Secrets structure created
[INFO] Step 4/10: Generating mTLS certificates...
[INFO] Generating mTLS certificates for observability cell
[INFO]   Observability Cell: obs-user-universe-main-dev
[INFO]   Components: prometheus,grafana,loki,tempo,otlp_collector,clickhouse,alertmanager
[INFO]   Creating certificate directory: /etc/fastorder/observability/certs/obs-user-universe-main-dev
[2026-01-12 20:35:59 UTC] USER=www-data EUID=0 PID=685342 ACTION=fsop ARGS=mkdir -p /etc/fastorder/observability/certs/obs-user-universe-main-dev
[2026-01-12 20:35:59 UTC] USER=www-data EUID=0 PID=685351 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-user-universe-main-dev
[INFO]   Generating CA certificate for obs-user-universe-main-dev
[2026-01-12 20:35:59 UTC] USER=www-data EUID=0 PID=685360 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-key.pem 4096
[2026-01-12 20:36:05 UTC] USER=www-data EUID=0 PID=685551 ACTION=fsop ARGS=openssl req -new -x509 -days 3650 -key /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-key.pem -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=obs-user-universe-main-dev-ca
[2026-01-12 20:36:06 UTC] USER=www-data EUID=0 PID=685563 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-key.pem
[2026-01-12 20:36:06 UTC] USER=www-data EUID=0 PID=685572 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem
[INFO]   CA certificate created: /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem
[INFO]   Generating certificate for: prometheus
[2026-01-12 20:36:06 UTC] USER=www-data EUID=0 PID=685581 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/prometheus-key.pem 2048
[2026-01-12 20:36:06 UTC] USER=www-data EUID=0 PID=685599 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-universe-main-dev/prometheus-key.pem -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/prometheus-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=prometheus.obs-user-universe-main-dev
[2026-01-12 20:36:06 UTC] USER=www-data EUID=0 PID=685608 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-universe-main-dev/prometheus-csr.pem -CA /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/prometheus-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = prometheus.obs-user-universe-main-dev
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-user-universe-main-dev/prometheus-cert.pem
[INFO]   Generating certificate for: grafana
[2026-01-12 20:36:06 UTC] USER=www-data EUID=0 PID=685647 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/grafana-key.pem 2048
[2026-01-12 20:36:06 UTC] USER=www-data EUID=0 PID=685656 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-universe-main-dev/grafana-key.pem -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/grafana-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=grafana.obs-user-universe-main-dev
[2026-01-12 20:36:06 UTC] USER=www-data EUID=0 PID=685665 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-universe-main-dev/grafana-csr.pem -CA /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/grafana-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = grafana.obs-user-universe-main-dev
[2026-01-12 20:36:06 UTC] USER=www-data EUID=0 PID=685674 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-user-universe-main-dev/grafana-key.pem
[2026-01-12 20:36:06 UTC] USER=www-data EUID=0 PID=685689 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-universe-main-dev/grafana-cert.pem
[2026-01-12 20:36:06 UTC] USER=www-data EUID=0 PID=685698 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-user-universe-main-dev/grafana-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-user-universe-main-dev/grafana-cert.pem
[INFO]   Generating certificate for: loki
[2026-01-12 20:36:06 UTC] USER=www-data EUID=0 PID=685707 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/loki-key.pem 2048
[2026-01-12 20:36:07 UTC] USER=www-data EUID=0 PID=685734 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-universe-main-dev/loki-key.pem -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/loki-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=loki.obs-user-universe-main-dev
[2026-01-12 20:36:07 UTC] USER=www-data EUID=0 PID=685743 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-universe-main-dev/loki-csr.pem -CA /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/loki-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = loki.obs-user-universe-main-dev
[2026-01-12 20:36:07 UTC] USER=www-data EUID=0 PID=685752 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-user-universe-main-dev/loki-key.pem
[2026-01-12 20:36:07 UTC] USER=www-data EUID=0 PID=685761 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-universe-main-dev/loki-cert.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-user-universe-main-dev/loki-cert.pem
[INFO]   Generating certificate for: tempo
[2026-01-12 20:36:07 UTC] USER=www-data EUID=0 PID=685779 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/tempo-key.pem 2048
[2026-01-12 20:36:07 UTC] USER=www-data EUID=0 PID=685788 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-universe-main-dev/tempo-key.pem -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/tempo-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=tempo.obs-user-universe-main-dev
[2026-01-12 20:36:07 UTC] USER=www-data EUID=0 PID=685800 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-universe-main-dev/tempo-csr.pem -CA /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/tempo-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = tempo.obs-user-universe-main-dev
[2026-01-12 20:36:08 UTC] USER=www-data EUID=0 PID=685818 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-universe-main-dev/tempo-cert.pem
[2026-01-12 20:36:08 UTC] USER=www-data EUID=0 PID=685827 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-user-universe-main-dev/tempo-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-user-universe-main-dev/tempo-cert.pem
[INFO]   Generating certificate for: otlp_collector
[2026-01-12 20:36:08 UTC] USER=www-data EUID=0 PID=685836 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/otlp_collector-key.pem 2048
[2026-01-12 20:36:08 UTC] USER=www-data EUID=0 PID=685848 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-universe-main-dev/otlp_collector-key.pem -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/otlp_collector-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=otlp_collector.obs-user-universe-main-dev
[2026-01-12 20:36:08 UTC] USER=www-data EUID=0 PID=685857 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-universe-main-dev/otlp_collector-csr.pem -CA /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/otlp_collector-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = otlp_collector.obs-user-universe-main-dev
[2026-01-12 20:36:08 UTC] USER=www-data EUID=0 PID=685877 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-user-universe-main-dev/otlp_collector-key.pem
[2026-01-12 20:36:08 UTC] USER=www-data EUID=0 PID=685940 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-user-universe-main-dev/otlp_collector-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-user-universe-main-dev/otlp_collector-cert.pem
[INFO]   Generating certificate for: clickhouse
[2026-01-12 20:36:08 UTC] USER=www-data EUID=0 PID=685970 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/clickhouse-key.pem 2048
[2026-01-12 20:36:09 UTC] USER=www-data EUID=0 PID=686032 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-universe-main-dev/clickhouse-key.pem -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/clickhouse-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=clickhouse.obs-user-universe-main-dev
[2026-01-12 20:36:09 UTC] USER=www-data EUID=0 PID=686042 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-universe-main-dev/clickhouse-csr.pem -CA /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/clickhouse-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = clickhouse.obs-user-universe-main-dev
[2026-01-12 20:36:09 UTC] USER=www-data EUID=0 PID=686051 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-user-universe-main-dev/clickhouse-key.pem
[2026-01-12 20:36:09 UTC] USER=www-data EUID=0 PID=686060 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-universe-main-dev/clickhouse-cert.pem
[2026-01-12 20:36:09 UTC] USER=www-data EUID=0 PID=686069 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-user-universe-main-dev/clickhouse-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-user-universe-main-dev/clickhouse-cert.pem
[INFO]   Generating certificate for: alertmanager
[2026-01-12 20:36:09 UTC] USER=www-data EUID=0 PID=686078 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/alertmanager-key.pem 2048
[2026-01-12 20:36:10 UTC] USER=www-data EUID=0 PID=686094 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-universe-main-dev/alertmanager-key.pem -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/alertmanager-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=alertmanager.obs-user-universe-main-dev
[2026-01-12 20:36:10 UTC] USER=www-data EUID=0 PID=686103 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-universe-main-dev/alertmanager-csr.pem -CA /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/alertmanager-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = alertmanager.obs-user-universe-main-dev
[2026-01-12 20:36:10 UTC] USER=www-data EUID=0 PID=686114 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-user-universe-main-dev/alertmanager-key.pem
[2026-01-12 20:36:10 UTC] USER=www-data EUID=0 PID=686123 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-universe-main-dev/alertmanager-cert.pem
[2026-01-12 20:36:10 UTC] USER=www-data EUID=0 PID=686140 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-user-universe-main-dev/alertmanager-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-user-universe-main-dev/alertmanager-cert.pem
[INFO]   Generating PHP client certificate for metrics service...
[2026-01-12 20:36:10 UTC] USER=www-data EUID=0 PID=686152 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/php-client-key.pem 2048
[2026-01-12 20:36:10 UTC] USER=www-data EUID=0 PID=686163 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-universe-main-dev/php-client-key.pem -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/php-client-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Dashboard/CN=php-metrics-client.obs-user-universe-main-dev
[2026-01-12 20:36:10 UTC] USER=www-data EUID=0 PID=686172 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-universe-main-dev/php-client-csr.pem -CA /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/php-client-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Dashboard, CN = php-metrics-client.obs-user-universe-main-dev
[2026-01-12 20:36:10 UTC] USER=www-data EUID=0 PID=686190 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-universe-main-dev/php-client-cert.pem
[2026-01-12 20:36:10 UTC] USER=www-data EUID=0 PID=686199 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-user-universe-main-dev/php-client-key.pem
[2026-01-12 20:36:10 UTC] USER=www-data EUID=0 PID=686208 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-user-universe-main-dev/php-client-cert.pem
[2026-01-12 20:36:10 UTC] USER=www-data EUID=0 PID=686217 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-user-universe-main-dev/php-client-csr.pem
[INFO]   PHP client certificate created: /etc/fastorder/observability/certs/obs-user-universe-main-dev/php-client-cert.pem
[INFO]   Generating Apache client certificate for mTLS reverse proxy...
[2026-01-12 20:36:11 UTC] USER=www-data EUID=0 PID=686226 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/apache-client-key.pem 2048
[2026-01-12 20:36:11 UTC] USER=www-data EUID=0 PID=686235 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-universe-main-dev/apache-client-key.pem -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/apache-client-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=ReverseProxy/CN=apache-proxy.obs-user-universe-main-dev
[2026-01-12 20:36:11 UTC] USER=www-data EUID=0 PID=686244 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-universe-main-dev/apache-client-csr.pem -CA /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-universe-main-dev/apache-client-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = ReverseProxy, CN = apache-proxy.obs-user-universe-main-dev
[2026-01-12 20:36:11 UTC] USER=www-data EUID=0 PID=686280 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-user-universe-main-dev/apache-client-combined.pem
[2026-01-12 20:36:11 UTC] USER=www-data EUID=0 PID=686289 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-universe-main-dev/apache-client-cert.pem
[2026-01-12 20:36:11 UTC] USER=www-data EUID=0 PID=686298 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-user-universe-main-dev/apache-client-key.pem
[2026-01-12 20:36:11 UTC] USER=www-data EUID=0 PID=686307 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-user-universe-main-dev/apache-client-cert.pem
[INFO]   Apache client certificate created: /etc/fastorder/observability/certs/obs-user-universe-main-dev/apache-client-cert.pem
[INFO]   Apache combined cert+key: /etc/fastorder/observability/certs/obs-user-universe-main-dev/apache-client-combined.pem
[INFO]   Storing mTLS certificates in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/user/universe/main/dev/mtls/php-client-UiuYG7",
    "Name": "fastorder/observability/user/universe/main/dev/mtls/php-client",
    "VersionId": "e4b01106-c087-4e5a-926c-95b9e1ce64d0"
}
[INFO]   mTLS certificates stored in Secrets Manager: fastorder/observability/user/universe/main/dev/mtls/php-client
[INFO] mTLS certificates generated successfully
[INFO]   Certificate directory: /etc/fastorder/observability/certs/obs-user-universe-main-dev
[INFO]   PHP client cert: /etc/fastorder/observability/certs/obs-user-universe-main-dev/php-client-cert.pem
[INFO]   PHP client key: /etc/fastorder/observability/certs/obs-user-universe-main-dev/php-client-key.pem
[INFO]   Apache client cert: /etc/fastorder/observability/certs/obs-user-universe-main-dev/apache-client-cert.pem
[INFO]   Apache combined (for SSLProxyMachineCertificateFile): /etc/fastorder/observability/certs/obs-user-universe-main-dev/apache-client-combined.pem
[OK]   mTLS certificates generated
[INFO] Step 5/10: Deploying log storage backend...
[INFO]   Provider: clickhouse (selected)
[INFO]   Note: Deployed before telemetry (OtelCol depends on log storage)
[INFO]   FQDN: logstore-user-universe-main-dev-clickhouse.fastorder.com
[INFO]   IP: 10.100.1.223
[INFO] Deploying log backend: clickhouse...
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] LOG STORAGE BACKEND DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: clickhouse
[INFO] Observability Cell: obs-user-universe-main-dev
[INFO] FQDN: logstore-user-universe-main-dev-clickhouse.fastorder.com
[INFO] IP: 10.100.1.223
[INFO] S3 Bucket: fastorder-logs-universe-dev
[INFO] Retention: 90 days
[INFO] ═══════════════════════════════════════════════════════════════

[2026-01-12 20:36:14 UTC] USER=unknown EUID=33 PID=686462 ACTION=fsop ARGS=chmod +x /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh
[INFO] Using provider: clickhouse
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh

[INFO] Executing provider deployment script...
/bin/chmod: changing permissions of '/opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh': Operation not permitted
[INFO] Parsed: SERVICE=user, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-user-universe-main-dev...
[2026-01-12 20:36:14 UTC] USER=www-data EUID=0 PID=686479 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-12 20:36:14 UTC] USER=www-data EUID=0 PID=686488 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-12 20:36:14 UTC] USER=www-data EUID=0 PID=686497 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-user-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-user-universe-main-dev
[INFO] IP Address: 10.100.1.223
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service clickhouse-server@obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server@obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service prometheus-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service grafana-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service tempo-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service alertmanager-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Found 24 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.223...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[INFO] Binding ClickHouse to allocated IP: 10.100.1.223
[INFO] Deploying ClickHouse for obs-user-universe-main-dev
[INFO]   FQDN: logstore-user-universe-main-dev-clickhouse.fastorder.com
[INFO]   Allocated IP: 10.100.1.223
[INFO]   VM IP: 10.100.1.223
[INFO]   Ports: HTTP=8123 TCP=9000 Interserver=9009
[INFO]   S3 Bucket: fastorder-logs-universe-dev (region=me-central-1)
[INFO]   Retention: 90 days
[INFO] Checking if ClickHouse is installed...
[OK]   ClickHouse already installed
[2026-01-12 20:36:15 UTC] USER=www-data EUID=0 PID=686725 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-server-obs-user-universe-main-dev/config.d
[2026-01-12 20:36:15 UTC] USER=www-data EUID=0 PID=686734 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-server-obs-user-universe-main-dev/users.d
[2026-01-12 20:36:15 UTC] USER=www-data EUID=0 PID=686743 ACTION=fsop ARGS=mkdir -p /var/lib/clickhouse-obs-user-universe-main-dev
[2026-01-12 20:36:15 UTC] USER=www-data EUID=0 PID=686752 ACTION=fsop ARGS=mkdir -p /var/log/clickhouse-server-obs-user-universe-main-dev
[2026-01-12 20:36:15 UTC] USER=www-data EUID=0 PID=686761 ACTION=passthru ARGS=chmod 755 /etc/clickhouse-server-obs-user-universe-main-dev
[2026-01-12 20:36:15 UTC] USER=www-data EUID=0 PID=686770 ACTION=passthru ARGS=chmod 700 /var/lib/clickhouse-obs-user-universe-main-dev
[2026-01-12 20:36:15 UTC] USER=www-data EUID=0 PID=686779 ACTION=passthru ARGS=chmod 750 /var/log/clickhouse-server-obs-user-universe-main-dev
[INFO] Found existing logs_writer credentials in Secrets Manager - reusing to maintain sync
[INFO] Found existing metrics_reader credentials in Secrets Manager - reusing to maintain sync
[INFO] TLS configuration exported for clickhouse
[INFO]   Cert: /etc/fastorder/observability/certs/obs-user-universe-main-dev/clickhouse-cert.pem
[INFO]   Key:  /etc/fastorder/observability/certs/obs-user-universe-main-dev/clickhouse-key.pem
[INFO]   CA:   /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem
[INFO] Configuring certificate permissions for clickhouse (user: clickhouse)
[INFO] Initializing certificate directory for obs-user-universe-main-dev...
[2026-01-12 20:36:18 UTC] USER=www-data EUID=0 PID=686865 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-12 20:36:18 UTC] USER=www-data EUID=0 PID=686874 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-12 20:36:18 UTC] USER=www-data EUID=0 PID=686883 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-12 20:36:18 UTC] USER=www-data EUID=0 PID=686892 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-user-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-user-universe-main-dev
[INFO]   Setting file permissions...
[2026-01-12 20:36:18 UTC] USER=www-data EUID=0 PID=686902 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-universe-main-dev/clickhouse-cert.pem
[2026-01-12 20:36:18 UTC] USER=www-data EUID=0 PID=686911 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem
[2026-01-12 20:36:18 UTC] USER=www-data EUID=0 PID=686920 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-user-universe-main-dev/clickhouse-key.pem
[INFO]   Setting file ownership...
[2026-01-12 20:36:18 UTC] USER=www-data EUID=0 PID=686929 ACTION=passthru ARGS=chown root:clickhouse /etc/fastorder/observability/certs/obs-user-universe-main-dev/clickhouse-key.pem
[2026-01-12 20:36:18 UTC] USER=www-data EUID=0 PID=686938 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-user-universe-main-dev/clickhouse-cert.pem /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for clickhouse
[INFO] Creating ClickHouse configuration...
[2026-01-12 20:36:18 UTC] USER=www-data EUID=0 PID=686986 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /etc/clickhouse-server-obs-user-universe-main-dev
[2026-01-12 20:36:18 UTC] USER=www-data EUID=0 PID=686996 ACTION=passthru ARGS=bash -c chmod 640 /etc/clickhouse-server-obs-user-universe-main-dev/*.xml
[OK]   ClickHouse configuration created
[INFO] Creating logs table schema...
[2026-01-12 20:36:19 UTC] USER=www-data EUID=0 PID=687014 ACTION=passthru ARGS=sed -i s/__RETENTION_DAYS__/90/g /etc/clickhouse-server-obs-user-universe-main-dev/logs_schema.sql
[2026-01-12 20:36:19 UTC] USER=www-data EUID=0 PID=687023 ACTION=passthru ARGS=chmod 644 /etc/clickhouse-server-obs-user-universe-main-dev/logs_schema.sql
[OK]   Logs schema created
[INFO] Creating systemd service...
[2026-01-12 20:36:19 UTC] USER=www-data EUID=0 PID=687041 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /var/lib/clickhouse-obs-user-universe-main-dev
[2026-01-12 20:36:19 UTC] USER=www-data EUID=0 PID=687051 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /var/log/clickhouse-server-obs-user-universe-main-dev
[2026-01-12 20:36:19 UTC] USER=www-data EUID=0 PID=687064 ACTION=passthru ARGS=chmod 700 /var/lib/clickhouse-obs-user-universe-main-dev
[OK]   Systemd service created
[INFO] Starting ClickHouse service...
[2026-01-12 20:36:19 UTC] USER=www-data EUID=0 PID=687073 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-12 20:36:20 UTC] USER=www-data EUID=0 PID=687138 ACTION=passthru ARGS=systemctl enable clickhouse-server-obs-user-universe-main-dev.service
[2026-01-12 20:36:20 UTC] USER=www-data EUID=0 PID=687194 ACTION=passthru ARGS=systemctl start clickhouse-server-obs-user-universe-main-dev.service
[INFO] Waiting for ClickHouse to be ready...
[OK]   ClickHouse is ready
[INFO] Initializing database schema...
[OK]   Schema initialized
[INFO] Storing ClickHouse credentials in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/user/universe/main/dev/clickhouse/server/logs_writer-ydMIzT",
    "Name": "fastorder/observability/user/universe/main/dev/clickhouse/server/logs_writer",
    "VersionId": "171009e0-a340-45ef-8bb4-5fbe48b93564"
}
[OK]   logs_writer credentials stored and verified in Secrets Manager
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/user/universe/main/dev/clickhouse/server/metrics_reader-HMOIKS",
    "Name": "fastorder/observability/user/universe/main/dev/clickhouse/server/metrics_reader",
    "VersionId": "78c5cff7-fdda-4b29-b129-d47d51b2ff1b"
}
[OK]   metrics_reader credentials stored and verified in Secrets Manager
[INFO] Validating ClickHouse deployment...
[INFO] ClickHouse version: 25.10.1.3832
[INFO] Tables created: .inner_id.68d8aade-b364-4fa6-a30b-6a4acf271412
.inner_id.c38a2486-cbb5-4960-93c8-75b75697c2b7
application_logs
error_logs_mv
iam_audit_event
metrics_all
metrics_all_exponential_histogram
metrics_all_gauge
metrics_all_histogram
metrics_all_sum
metrics_all_summary
otel_logs
request_logs_mv
security_access
traces_all
traces_all_trace_id_ts
traces_all_trace_id_ts_mv
[INFO] Test log inserted. Total logs: 5
[OK]   ✅ ClickHouse deployment validated
[INFO] Setting up clickhouse-backup for backup management...
[OK]   clickhouse-backup already installed
[INFO] Creating clickhouse-backup configuration...
[2026-01-12 20:36:31 UTC] USER=www-data EUID=0 PID=687472 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-backup
[2026-01-12 20:36:31 UTC] USER=www-data EUID=0 PID=687481 ACTION=passthru ARGS=chown root:clickhouse /etc/clickhouse-backup
[2026-01-12 20:36:31 UTC] USER=www-data EUID=0 PID=687490 ACTION=passthru ARGS=chmod 750 /etc/clickhouse-backup
[2026-01-12 20:36:31 UTC] USER=www-data EUID=0 PID=687508 ACTION=passthru ARGS=chown root:clickhouse /etc/clickhouse-backup/config-obs-user-universe-main-dev.yml
[2026-01-12 20:36:31 UTC] USER=www-data EUID=0 PID=687517 ACTION=passthru ARGS=chmod 640 /etc/clickhouse-backup/config-obs-user-universe-main-dev.yml
[2026-01-12 20:36:31 UTC] USER=www-data EUID=0 PID=687553 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-12 20:36:31 UTC] USER=www-data EUID=0 PID=687604 ACTION=passthru ARGS=systemctl enable clickhouse-backup-api-obs-user-universe-main-dev.service
[2026-01-12 20:36:32 UTC] USER=www-data EUID=0 PID=687653 ACTION=passthru ARGS=systemctl start clickhouse-backup-api-obs-user-universe-main-dev.service
[2026-01-12 20:36:32 UTC] USER=www-data EUID=0 PID=687679 ACTION=passthru ARGS=systemctl enable clickhouse-backup@obs-user-universe-main-dev.timer
[2026-01-12 20:36:32 UTC] USER=www-data EUID=0 PID=687805 ACTION=passthru ARGS=systemctl start clickhouse-backup@obs-user-universe-main-dev.timer
[OK]   clickhouse-backup configured and started
[INFO] Setting up ClickHouse exporter for Prometheus...
[OK]   clickhouse_exporter already installed
[2026-01-12 20:36:33 UTC] USER=www-data EUID=0 PID=687839 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-12 20:36:33 UTC] USER=www-data EUID=0 PID=687903 ACTION=passthru ARGS=systemctl enable clickhouse_exporter-obs-user-universe-main-dev.service
[2026-01-12 20:36:33 UTC] USER=www-data EUID=0 PID=687966 ACTION=passthru ARGS=systemctl start clickhouse_exporter-obs-user-universe-main-dev.service
[OK]   clickhouse_exporter configured and started

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ ClickHouse Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   FQDN: logstore-user-universe-main-dev-clickhouse.fastorder.com
[INFO]   IP: 10.100.1.223
[INFO]   HTTP Port: 8123
[INFO]   Native Port: 9000
[INFO]   Database: logs
[INFO]   Retention: 90 days
[INFO]   Storage: Tiered (Local → S3: fastorder-logs-universe-dev in me-central-1)
[INFO] 
[INFO] Backup & Monitoring:
[INFO]   clickhouse-backup API: http://10.100.1.223:7171
[INFO]   clickhouse_exporter: http://10.100.1.223:9116/metrics
[INFO]   Backup Schedule: Daily at 2:00 AM
[INFO]   Local Backups Retained: 7
[INFO] 
[INFO] Credentials stored in AWS Secrets Manager:
[INFO]   Writers: fastorder/observability/user/universe/main/dev/clickhouse/server/logs_writer
[INFO]   Readers: fastorder/observability/user/universe/main/dev/clickhouse/server/metrics_reader (for PHP metrics service)
[INFO] 
[INFO] Example queries (using credentials from Secrets Manager):
[INFO]   # Write logs:
[INFO]   clickhouse-client --host logstore-user-universe-main-dev-clickhouse.fastorder.com --port 9000 --user logs_writer --password '***' --query 'SELECT 1'
[INFO] 
[INFO]   # Read metrics (PHP metrics service):
[INFO]   clickhouse-client --host logstore-user-universe-main-dev-clickhouse.fastorder.com --port 9000 --user metrics_reader --password '***' --query 'SELECT * FROM system.metrics'
[INFO] 
[INFO] HTTPS Setup (run on web-03/skeleton server):
[INFO]   # Set up HTTPS reverse proxy with Let's Encrypt:
[INFO]   OBS_CELL=obs-user-universe-main-dev BACKEND_IP=10.100.1.223 sudo bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/../https/setup-clickhouse-https.sh
[INFO] 
[INFO]   # Or add --setup-https flag when running this script
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Log Storage Backend Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: clickhouse
[INFO] FQDN: logstore-user-universe-main-dev-clickhouse.fastorder.com
[INFO] IP: 10.100.1.223
[INFO] Retention: 90 days
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering ClickHouse in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       ClickHouse
[INFO]   Identifier:        user-universe-main-dev-clickhouse
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.223
[INFO]   Port:              8443
[INFO]   FQDN:              logstore-user-universe-main-dev-clickhouse.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-universe-main-dev (service=user, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ INVALID REQUEST
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] Response: {"success":false,"error":"Invalid JSON: Control character error, possibly incorrectly encoded"}
[ERROR] 
[ERROR] Request payload:
  {
    "env_id": "user-universe-main-dev",
    "application": "ClickHouse",
    "identifier": "user-universe-main-dev-clickhouse",
    "identifier_parent": "cluster",
    "ip": "10.100.1.223",
    "port": 8443,
    "fqdn": "logstore-user-universe-main-dev-clickhouse.fastorder.com",
    "status": "running",
    "meta": {
      "role": "log_storage",
      "provider": "clickhouse",
      "version": "25.10
  1.3832",
      "http_port": 8123,
      "native_port": 9000,
      "https_port": 8443,
      "protocol": "https",
      "metrics_enabled": true,
      "metrics_port": 8123,
      "metrics_path": "/metrics",
      "health_endpoint": "https://logstore-user-universe-main-dev-clickhouse.fastorder.com/ping",
      "retention_days": 90,
      "s3_bucket": "fastorder-logs-universe-dev"
  }
  }
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[WARN] ⚠️  Failed to register ClickHouse (service is running)
[OK]   clickhouse deployed successfully
[OK]   Log storage backend deployed
[INFO] Step 6/10: Deploying telemetry collector...
[INFO]   Provider: otlp (backend implementation - internal)
[INFO]   Endpoint: telemetry-user-universe-main-dev-opentelemetry.fastorder.com (stable, exposed to clients)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] TELEMETRY COLLECTOR DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: otlp
[INFO] Observability Cell: obs-user-universe-main-dev
[INFO] FQDN: telemetry-user-universe-main-dev-opentelemetry.fastorder.com
[INFO] IP: 10.100.1.226
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: otlp
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Telemetry/provider/otlp.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=user, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-user-universe-main-dev...
[2026-01-12 20:36:37 UTC] USER=www-data EUID=0 PID=688085 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-12 20:36:37 UTC] USER=www-data EUID=0 PID=688106 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-12 20:36:37 UTC] USER=www-data EUID=0 PID=688117 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-12 20:36:37 UTC] USER=www-data EUID=0 PID=688135 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-user-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-user-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-user-universe-main-dev
[INFO] IP Address: 10.100.1.226
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service clickhouse-server@obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server@obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service prometheus-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service grafana-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service tempo-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service alertmanager-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Found 24 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.226...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[INFO] Binding to allocated IP: 10.100.1.226
[INFO] Deploying OpenTelemetry Collector for observability cell: obs-user-universe-main-dev
[INFO] FQDN:         telemetry-user-universe-main-dev-opentelemetry.fastorder.com
[INFO] Allocated IP: 10.100.1.226
[INFO] VM IP:        10.100.1.226
[INFO] Ports:        gRPC=4317 HTTP=4318 Metrics=8888 Prom=8889
[OK]   User 'otelcol' already exists
[INFO] Checking if OpenTelemetry Collector is installed...
[OK]   OpenTelemetry Collector already installed at /usr/local/bin/otelcol-contrib
[INFO] Creating configuration/data directories...
[2026-01-12 20:36:38 UTC] USER=www-data EUID=0 PID=688359 ACTION=passthru ARGS=mkdir -p /etc/otelcol/obs-user-universe-main-dev
[2026-01-12 20:36:38 UTC] USER=www-data EUID=0 PID=688377 ACTION=passthru ARGS=chown -R otelcol:otelcol /etc/otelcol/obs-user-universe-main-dev /var/lib/otelcol/obs-user-universe-main-dev
[2026-01-12 20:36:38 UTC] USER=www-data EUID=0 PID=688386 ACTION=passthru ARGS=chmod 0750 /etc/otelcol/obs-user-universe-main-dev
[2026-01-12 20:36:38 UTC] USER=www-data EUID=0 PID=688395 ACTION=passthru ARGS=chmod 0750 /var/lib/otelcol/obs-user-universe-main-dev
[INFO] Retrieving ClickHouse credentials from Secrets Manager...
[OK]   Retrieved ClickHouse credentials from Secrets Manager
[INFO] Creating OpenTelemetry Collector configuration...
[INFO] ClickHouse exporter enabled: tcp://logstore-user-universe-main-dev-clickhouse.fastorder.com:9000
[2026-01-12 20:36:40 UTC] USER=www-data EUID=0 PID=688497 ACTION=passthru ARGS=chown otelcol:otelcol /etc/otelcol/obs-user-universe-main-dev/config.yaml
[2026-01-12 20:36:40 UTC] USER=www-data EUID=0 PID=688506 ACTION=passthru ARGS=chmod 0640 /etc/otelcol/obs-user-universe-main-dev/config.yaml
[OK]   Configuration created at /etc/otelcol/obs-user-universe-main-dev/config.yaml
[INFO] Setting up TLS certificate permissions...
[INFO] Configuring certificate permissions for otlp_collector (user: otelcol)
[INFO] Initializing certificate directory for obs-user-universe-main-dev...
[2026-01-12 20:36:40 UTC] USER=www-data EUID=0 PID=688515 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-12 20:36:40 UTC] USER=www-data EUID=0 PID=688534 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-12 20:36:40 UTC] USER=www-data EUID=0 PID=688543 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-user-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-user-universe-main-dev
[INFO]   Setting file permissions...
[2026-01-12 20:36:40 UTC] USER=www-data EUID=0 PID=688553 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-universe-main-dev/otlp_collector-cert.pem
[2026-01-12 20:36:40 UTC] USER=www-data EUID=0 PID=688571 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-user-universe-main-dev/otlp_collector-key.pem
[INFO]   Setting file ownership...
[2026-01-12 20:36:40 UTC] USER=www-data EUID=0 PID=688582 ACTION=passthru ARGS=chown root:otelcol /etc/fastorder/observability/certs/obs-user-universe-main-dev/otlp_collector-key.pem
[2026-01-12 20:36:40 UTC] USER=www-data EUID=0 PID=688594 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-user-universe-main-dev/otlp_collector-cert.pem /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for otlp_collector
[OK]   Certificate permissions configured
[INFO] Creating systemd service: otelcol-obs-user-universe-main-dev
[OK]   Systemd service created at /etc/systemd/system/otelcol-obs-user-universe-main-dev.service
[INFO] Adding /etc/hosts entry for telemetry-user-universe-main-dev-opentelemetry.fastorder.com -> 10.100.1.226
[2026-01-12 20:36:40 UTC] USER=www-data EUID=0 PID=688621 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*telemetry-user-universe-main-dev-opentelemetry.fastorder.com/10.100.1.226    telemetry-user-universe-main-dev-opentelemetry.fastorder.com/ /etc/hosts
[OK]   Updated /etc/hosts entry to use VM_IP
[INFO] Storing OTLP configuration metadata in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/user/universe/main/dev/otlp/collector-8o5H25",
    "Name": "fastorder/observability/user/universe/main/dev/otlp/collector",
    "VersionId": "98470a5a-ff40-4ca3-b15a-051f20471e2e"
}
[OK]   Configuration metadata stored/updated in AWS Secrets Manager: fastorder/observability/user/universe/main/dev/otlp/collector
[INFO] Enabling and starting OpenTelemetry Collector service...
[2026-01-12 20:36:43 UTC] USER=www-data EUID=0 PID=688676 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-12 20:36:43 UTC] USER=www-data EUID=0 PID=688730 ACTION=passthru ARGS=systemctl enable otelcol-obs-user-universe-main-dev.service
[2026-01-12 20:36:44 UTC] USER=www-data EUID=0 PID=688776 ACTION=passthru ARGS=systemctl restart otelcol-obs-user-universe-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-01-12 20:36:47 UTC] USER=www-data EUID=0 PID=688861 ACTION=passthru ARGS=systemctl is-active --quiet otelcol-obs-user-universe-main-dev.service
[OK]   ✅ OpenTelemetry Collector is running
[OK]   ✅ gRPC endpoint listening on port 4317
[OK]   ✅ HTTP endpoint listening on port 4318
[OK]   ✅ Prometheus metrics endpoint listening on port 8889
[INFO] Service logs (last 10 lines):
[2026-01-12 20:36:47 UTC] USER=www-data EUID=0 PID=688877 ACTION=passthru ARGS=journalctl -u otelcol-obs-user-universe-main-dev.service -n 10 --no-pager
Jan 12 20:36:44 web-03 otelcol-obs-user-universe-main-dev[688785]: 2026-01-12T20:36:44.342Z        info        internal/resourcedetection.go:125        began detecting resource information        {"kind": "processor", "name": "resourcedetection", "pipeline": "logs"}
Jan 12 20:36:44 web-03 otelcol-obs-user-universe-main-dev[688785]: 2026-01-12T20:36:44.346Z        info        system/system.go:201        This attribute changed from int to string. Temporarily switch back to int using the feature gate.        {"kind": "processor", "name": "resourcedetection", "pipeline": "logs", "attribute": "host.cpu.family", "feature gate": "processor.resourcedetection.hostCPUModelAndFamilyAsString"}
Jan 12 20:36:44 web-03 otelcol-obs-user-universe-main-dev[688785]: 2026-01-12T20:36:44.347Z        info        system/system.go:220        This attribute changed from int to string. Temporarily switch back to int using the feature gate.        {"kind": "processor", "name": "resourcedetection", "pipeline": "logs", "attribute": "host.cpu.model.id", "feature gate": "processor.resourcedetection.hostCPUModelAndFamilyAsString"}
Jan 12 20:36:44 web-03 otelcol-obs-user-universe-main-dev[688785]: 2026-01-12T20:36:44.347Z        info        internal/resourcedetection.go:139        detected resource information        {"kind": "processor", "name": "resourcedetection", "pipeline": "logs", "resource": {"host.name":"web-03","os.type":"linux"}}
Jan 12 20:36:44 web-03 otelcol-obs-user-universe-main-dev[688785]: 2026-01-12T20:36:44.348Z        info        otlpreceiver@v0.91.0/otlp.go:83        Starting GRPC server        {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "10.100.1.226:4317"}
Jan 12 20:36:44 web-03 otelcol-obs-user-universe-main-dev[688785]: 2026-01-12T20:36:44.348Z        info        otlpreceiver@v0.91.0/otlp.go:101        Starting HTTP server        {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "10.100.1.226:4318"}
Jan 12 20:36:44 web-03 otelcol-obs-user-universe-main-dev[688785]: 2026-01-12T20:36:44.364Z        info        prometheusreceiver@v0.91.0/metrics_receiver.go:240        Starting discovery manager        {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
Jan 12 20:36:44 web-03 otelcol-obs-user-universe-main-dev[688785]: 2026-01-12T20:36:44.364Z        info        prometheusreceiver@v0.91.0/metrics_receiver.go:231        Scrape job added        {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "jobName": "otel-collector"}
Jan 12 20:36:44 web-03 otelcol-obs-user-universe-main-dev[688785]: 2026-01-12T20:36:44.365Z        info        service@v0.91.0/service.go:171        Everything is ready. Begin running and processing data.
Jan 12 20:36:44 web-03 otelcol-obs-user-universe-main-dev[688785]: 2026-01-12T20:36:44.365Z        info        prometheusreceiver@v0.91.0/metrics_receiver.go:282        Starting scrape manager        {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Telemetry Collector Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: otlp
[INFO] FQDN: telemetry-user-universe-main-dev-opentelemetry.fastorder.com
[INFO] IP: 10.100.1.226
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering OpenTelemetry Collector in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       OpenTelemetry Collector
[INFO]   Identifier:        user-universe-main-dev-opentelemetry
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.226
[INFO]   Port:              4317
[INFO]   FQDN:              telemetry-user-universe-main-dev-opentelemetry.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-universe-main-dev (service=user, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: d78306cd-7efc-44a7-856d-9c09820cb486
[SUCCESS] Environment UUID: b683e8be-3b47-4d33-9d5c-389c63a0ae74
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b683e8be-3b47-4d33-9d5c-389c63a0ae74
[OK]   ✅ OpenTelemetry Collector registered in dashboard
[INFO] Setting up OpenTelemetry Collector metrics collection timer...
[2026-01-12 20:36:47 UTC] USER=www-data EUID=0 PID=688943 ACTION=passthru ARGS=mv /tmp/otelcol-metrics-user-universe-main-dev.service /etc/systemd/system/
[2026-01-12 20:36:47 UTC] USER=www-data EUID=0 PID=688952 ACTION=passthru ARGS=mv /tmp/otelcol-metrics-user-universe-main-dev.timer /etc/systemd/system/
[2026-01-12 20:36:47 UTC] USER=www-data EUID=0 PID=688961 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-12 20:36:48 UTC] USER=www-data EUID=0 PID=689005 ACTION=passthru ARGS=systemctl enable otelcol-metrics-user-universe-main-dev.timer
[2026-01-12 20:36:48 UTC] USER=www-data EUID=0 PID=689051 ACTION=passthru ARGS=systemctl start otelcol-metrics-user-universe-main-dev.timer
[OK]   ✅ Metrics collection timer installed and started
[OK]   Telemetry collector (otlp) deployed successfully

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Step 7/10: METRICS BACKEND DEPLOYMENT
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Provider: prometheus
[INFO]   OBS Cell: obs-user-universe-main-dev
[INFO]   FQDN: metrics-user-universe-main-dev-prometheus.fastorder.com
[INFO]   IP: 10.100.1.221
[INFO]   Script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/deploy-metrics.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📊 METRICS DEPLOYMENT WRAPPER STARTED
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Script: deploy-metrics.sh
[INFO] Timestamp: 2026-01-12 20:36:48 UTC
[INFO] Arguments: --provider prometheus --obs-cell obs-user-universe-main-dev --fqdn metrics-user-universe-main-dev-prometheus.fastorder.com --ip 10.100.1.221

[INFO] ═══════════════════════════════════════════════════════════════
[INFO] METRICS DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: prometheus
[INFO] Observability Cell: obs-user-universe-main-dev
[INFO] FQDN: metrics-user-universe-main-dev-prometheus.fastorder.com
[INFO] IP: 10.100.1.221
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: prometheus
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/provider/prometheus.sh

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/provider/prometheus.sh
[INFO]   OBS_CELL: obs-user-universe-main-dev
[INFO]   FQDN: metrics-user-universe-main-dev-prometheus.fastorder.com
[INFO]   IP: 10.100.1.221
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Parsed: SERVICE=user, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-user-universe-main-dev...
[2026-01-12 20:36:48 UTC] USER=www-data EUID=0 PID=689079 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-12 20:36:49 UTC] USER=www-data EUID=0 PID=689098 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-12 20:36:49 UTC] USER=www-data EUID=0 PID=689107 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-user-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-user-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-user-universe-main-dev
[INFO] IP Address: 10.100.1.221
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service clickhouse-server@obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server@obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service prometheus-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service grafana-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service tempo-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service alertmanager-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Found 24 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.221...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[INFO] Deploying Prometheus for observability cell: obs-user-universe-main-dev
[INFO] FQDN: metrics-user-universe-main-dev-prometheus.fastorder.com
[INFO] IP: 10.100.1.221
[INFO] Prometheus Port: 9090
[INFO] Checking if Prometheus is installed...
[OK]   Prometheus already installed at /usr/local/bin/prometheus
[INFO] Checking if Node Exporter is installed...
[OK]   Node Exporter already installed at /usr/local/bin/node_exporter
[2026-01-12 20:36:49 UTC] USER=www-data EUID=0 PID=689325 ACTION=fsop ARGS=mkdir -p /etc/prometheus/obs-user-universe-main-dev
[INFO] Creating Node Exporter TLS web config...
[INFO] Creating Node Exporter systemd service with TLS...
[2026-01-12 20:36:49 UTC] USER=www-data EUID=0 PID=689352 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-12 20:36:50 UTC] USER=www-data EUID=0 PID=689399 ACTION=passthru ARGS=systemctl enable node_exporter-obs-user-universe-main-dev.service
[2026-01-12 20:36:50 UTC] USER=www-data EUID=0 PID=689453 ACTION=passthru ARGS=systemctl restart node_exporter-obs-user-universe-main-dev.service
[OK]   Node Exporter service configured and started
[INFO] Creating configuration directory: /etc/prometheus/obs-user-universe-main-dev
[2026-01-12 20:36:50 UTC] USER=www-data EUID=0 PID=689467 ACTION=fsop ARGS=mkdir -p /etc/prometheus/obs-user-universe-main-dev
[2026-01-12 20:36:50 UTC] USER=www-data EUID=0 PID=689478 ACTION=fsop ARGS=mkdir -p /var/lib/prometheus/obs-user-universe-main-dev
[2026-01-12 20:36:50 UTC] USER=www-data EUID=0 PID=689487 ACTION=fsop ARGS=mkdir -p /etc/prometheus/obs-user-universe-main-dev/rules
[INFO] Creating Prometheus configuration...
[INFO] Generated FQDNs:
[INFO]   Prometheus:   metrics-user-universe-main-dev-prometheus.fastorder.com
[INFO]   Alertmanager: alerts-user-universe-main-dev-alertmanager.fastorder.com
[INFO]   Grafana:      dashboards-user-universe-main-dev-grafana.fastorder.com
[INFO]   Otelcol:      telemetry-user-universe-main-dev-opentelemetry.fastorder.com
[OK]   Configuration created at /etc/prometheus/obs-user-universe-main-dev/prometheus.yml
[INFO] Creating Prometheus web config for HTTPS...
[OK]   Web config created at /etc/prometheus/obs-user-universe-main-dev/web-config.yml
[INFO] Creating basic alerting rules...
[OK]   Alerting rules created
[2026-01-12 20:36:50 UTC] USER=www-data EUID=0 PID=689523 ACTION=fsop ARGS=mkdir -p /etc/prometheus/obs-user-universe-main-dev/targets
[2026-01-12 20:36:50 UTC] USER=www-data EUID=0 PID=689532 ACTION=passthru ARGS=bash -c cat > '/etc/prometheus/obs-user-universe-main-dev/targets/.placeholder.yml' << 'EOF'
# Placeholder file to prevent file_sd_configs warning
# Application targets will be added here automatically
[]
EOF
[INFO] Creating systemd service: prometheus-obs-user-universe-main-dev
[INFO] Binding to: 10.100.1.221:9090
[OK]   Systemd service created
[INFO] Configuring certificate permissions...
[INFO] Configuring certificate permissions for prometheus (user: root)
[INFO] Initializing certificate directory for obs-user-universe-main-dev...
[2026-01-12 20:36:50 UTC] USER=www-data EUID=0 PID=689551 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-12 20:36:50 UTC] USER=www-data EUID=0 PID=689560 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-12 20:36:50 UTC] USER=www-data EUID=0 PID=689569 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-12 20:36:50 UTC] USER=www-data EUID=0 PID=689578 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-user-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-user-universe-main-dev
[INFO]   Setting file permissions...
[2026-01-12 20:36:50 UTC] USER=www-data EUID=0 PID=689588 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-universe-main-dev/prometheus-cert.pem
[2026-01-12 20:36:51 UTC] USER=www-data EUID=0 PID=689597 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem
[2026-01-12 20:36:51 UTC] USER=www-data EUID=0 PID=689617 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-user-universe-main-dev/prometheus-key.pem
[INFO]   Setting file ownership...
[2026-01-12 20:36:51 UTC] USER=www-data EUID=0 PID=689635 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-user-universe-main-dev/prometheus-cert.pem /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for prometheus
[OK]   Certificate permissions configured
[INFO] Adding /etc/hosts entry for metrics-user-universe-main-dev-prometheus.fastorder.com -> 10.100.1.221
[2026-01-12 20:36:51 UTC] USER=www-data EUID=0 PID=689646 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*metrics-user-universe-main-dev-prometheus.fastorder.com/10.100.1.221    metrics-user-universe-main-dev-prometheus.fastorder.com/ /etc/hosts
[OK]   Updated /etc/hosts entry to use VM_IP
[INFO] Validating Prometheus configuration...
Checking /etc/prometheus/obs-user-universe-main-dev/prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: /etc/prometheus/obs-user-universe-main-dev/prometheus.yml is valid prometheus config file syntax

Checking /etc/prometheus/obs-user-universe-main-dev/rules/basic_alerts.yml
  SUCCESS: 4 rules found

[OK]   ✅ Configuration is valid
[INFO] Storing Prometheus configuration in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/user/universe/main/dev/prometheus/server-i0UMLg",
    "Name": "fastorder/observability/user/universe/main/dev/prometheus/server",
    "VersionId": "abb704ed-4f90-482e-a184-4e6cd1191220"
}
[OK]   Configuration stored in AWS Secrets Manager
[INFO] Enabling and starting Prometheus service...
[2026-01-12 20:36:53 UTC] USER=www-data EUID=0 PID=689689 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-12 20:36:54 UTC] USER=www-data EUID=0 PID=689735 ACTION=passthru ARGS=systemctl enable prometheus-obs-user-universe-main-dev.service
[2026-01-12 20:36:55 UTC] USER=www-data EUID=0 PID=689807 ACTION=passthru ARGS=systemctl restart prometheus-obs-user-universe-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-01-12 20:36:58 UTC] USER=www-data EUID=0 PID=689882 ACTION=passthru ARGS=systemctl is-active --quiet prometheus-obs-user-universe-main-dev.service
[OK]   ✅ Prometheus is running
[OK]   ✅ Prometheus web interface listening on port 9090
[OK]   ✅ Prometheus health check passed (HTTPS)
[INFO] ═══════════════════════════════════════════════════════════════
[OK]   Prometheus Web UI: https://metrics-user-universe-main-dev-prometheus.fastorder.com:9090
[OK]   Targets: https://metrics-user-universe-main-dev-prometheus.fastorder.com:9090/targets
[OK]   Alerts: https://metrics-user-universe-main-dev-prometheus.fastorder.com:9090/alerts
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service logs (last 10 lines):
[2026-01-12 20:37:00 UTC] USER=www-data EUID=0 PID=689898 ACTION=passthru ARGS=journalctl -u prometheus-obs-user-universe-main-dev.service -n 10 --no-pager
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Provider script completed with exit code: 0
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Metrics Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: prometheus
[INFO] FQDN: metrics-user-universe-main-dev-prometheus.fastorder.com
[INFO] IP: 10.100.1.221
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering Prometheus in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Prometheus
[INFO]   Identifier:        user-universe-main-dev-prometheus
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.221
[INFO]   Port:              9090
[INFO]   FQDN:              metrics-user-universe-main-dev-prometheus.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-universe-main-dev (service=user, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 67fa341d-dd98-4a64-8495-5938360bcb4e
[SUCCESS] Environment UUID: b683e8be-3b47-4d33-9d5c-389c63a0ae74
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b683e8be-3b47-4d33-9d5c-389c63a0ae74
[OK]   Prometheus registered in dashboard

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Step 7/10: METRICS DEPLOYMENT RESULT
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Exit code: 0
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ Metrics backend (prometheus) deployed successfully
[INFO] Step 8/10: Deploying traces backend...
[INFO]   Provider: tempo (selected)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] TRACES DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: tempo
[INFO] Observability Cell: obs-user-universe-main-dev
[INFO] FQDN: traces-user-universe-main-dev-tempo.fastorder.com
[INFO] IP: 10.100.1.224
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: tempo
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Traces/provider/tempo.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=user, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-user-universe-main-dev...
[2026-01-12 20:37:00 UTC] USER=www-data EUID=0 PID=689973 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-12 20:37:00 UTC] USER=www-data EUID=0 PID=689983 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-12 20:37:00 UTC] USER=www-data EUID=0 PID=689992 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-12 20:37:00 UTC] USER=www-data EUID=0 PID=690002 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-user-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-user-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-user-universe-main-dev
[INFO] IP Address: 10.100.1.224
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service clickhouse-server@obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server@obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service prometheus-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service grafana-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service tempo-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service alertmanager-obs-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-user-universe-main-dev.service belongs to current cell (skipping)
[INFO] Found 24 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.224...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[INFO] Binding Tempo to allocated IP: 10.100.1.224
[INFO] Deploying Grafana Tempo for observability cell: obs-user-universe-main-dev
[INFO] FQDN: traces-user-universe-main-dev-tempo.fastorder.com
[INFO] Allocated IP: 10.100.1.224
[INFO] VM IP: 10.100.1.224
[INFO] Ports: HTTP=3200 gRPC=9319, OTLP gRPC=4317, OTLP HTTP=4318
[INFO] Checking if Grafana Tempo is installed...
[OK]   Grafana Tempo already installed at /usr/local/bin/tempo
[INFO] Preparing configuration and data directories...
[2026-01-12 20:37:01 UTC] USER=www-data EUID=0 PID=690236 ACTION=passthru ARGS=mkdir -p /etc/tempo/obs-user-universe-main-dev
[2026-01-12 20:37:01 UTC] USER=www-data EUID=0 PID=690254 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-user-universe-main-dev/wal
[2026-01-12 20:37:01 UTC] USER=www-data EUID=0 PID=690264 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-user-universe-main-dev/blocks
[2026-01-12 20:37:01 UTC] USER=www-data EUID=0 PID=690283 ACTION=passthru ARGS=chown -R tempo:tempo /etc/tempo/obs-user-universe-main-dev /var/lib/tempo/obs-user-universe-main-dev
[INFO] Creating Grafana Tempo configuration...
[INFO] TLS configuration exported for tempo
[INFO]   Cert: /etc/fastorder/observability/certs/obs-user-universe-main-dev/tempo-cert.pem
[INFO]   Key:  /etc/fastorder/observability/certs/obs-user-universe-main-dev/tempo-key.pem
[INFO]   CA:   /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem
[INFO] Setting up certificate permissions for Tempo...
[INFO] Configuring certificate permissions for tempo (user: tempo)
[INFO] Initializing certificate directory for obs-user-universe-main-dev...
[2026-01-12 20:37:01 UTC] USER=www-data EUID=0 PID=690310 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-12 20:37:01 UTC] USER=www-data EUID=0 PID=690319 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-12 20:37:01 UTC] USER=www-data EUID=0 PID=690328 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-12 20:37:01 UTC] USER=www-data EUID=0 PID=690337 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-user-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-user-universe-main-dev
[INFO]   Setting file permissions...
[2026-01-12 20:37:01 UTC] USER=www-data EUID=0 PID=690347 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-universe-main-dev/tempo-cert.pem
[2026-01-12 20:37:01 UTC] USER=www-data EUID=0 PID=690357 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem
[2026-01-12 20:37:01 UTC] USER=www-data EUID=0 PID=690366 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-user-universe-main-dev/tempo-key.pem
[INFO]   Setting file ownership...
[2026-01-12 20:37:01 UTC] USER=www-data EUID=0 PID=690375 ACTION=passthru ARGS=chown root:tempo /etc/fastorder/observability/certs/obs-user-universe-main-dev/tempo-key.pem
[2026-01-12 20:37:01 UTC] USER=www-data EUID=0 PID=690384 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-user-universe-main-dev/tempo-cert.pem /etc/fastorder/observability/certs/obs-user-universe-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for tempo
[2026-01-12 20:37:01 UTC] USER=www-data EUID=0 PID=690403 ACTION=passthru ARGS=chown tempo:tempo /etc/tempo/obs-user-universe-main-dev/config.yaml
[2026-01-12 20:37:01 UTC] USER=www-data EUID=0 PID=690412 ACTION=passthru ARGS=chmod 640 /etc/tempo/obs-user-universe-main-dev/config.yaml
[OK]   Configuration created at /etc/tempo/obs-user-universe-main-dev/config.yaml
[INFO] Creating systemd service: tempo-obs-user-universe-main-dev
[OK]   Systemd service created
[INFO] Adding /etc/hosts entry for traces-user-universe-main-dev-tempo.fastorder.com -> 10.100.1.224
[2026-01-12 20:37:01 UTC] USER=www-data EUID=0 PID=690431 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*traces-user-universe-main-dev-tempo.fastorder.com/10.100.1.224    traces-user-universe-main-dev-tempo.fastorder.com/ /etc/hosts
[OK]   Updated /etc/hosts entry to use VM_IP
[INFO] Storing Tempo configuration in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/user/universe/main/dev/tempo/server-ZxDhwq",
    "Name": "fastorder/observability/user/universe/main/dev/tempo/server",
    "VersionId": "fb9a9840-620f-427c-8355-336c143f2b30"
}
[OK]   Tempo configuration stored/updated in AWS Secrets Manager: fastorder/observability/user/universe/main/dev/tempo/server
[WARN] Port cleanup library not found, skipping automatic cleanup
[INFO] Adding iptables redirect for Tempo internal communication (required for search)...
[INFO] ╔════════════════════════════════════════════════════════════════════════╗
[INFO] ║  TEMPO IPTABLES DNAT CONFIGURATION (Audit Log)                        ║
[INFO] ╠════════════════════════════════════════════════════════════════════════╣
[INFO] ║  OBS_CELL:     obs-user-universe-main-dev
[INFO] ║  VM_IP:        10.100.1.224
[INFO] ║  GRPC_PORT:    9319 (unique: 9095 + last_octet)
[INFO] ║  TEMPO_UID:    989
[INFO] ║  TIMESTAMP:    2026-01-12T20:37:04Z
[INFO] ╚════════════════════════════════════════════════════════════════════════╝
[INFO] Using --uid-owner 989 for DNAT rule (scoped to tempo user)
[2026-01-12 20:37:04 UTC] USER=www-data EUID=0 PID=690502 ACTION=passthru ARGS=iptables -t nat -A OUTPUT -p tcp -d 127.0.0.1 --dport 9319 -m owner --uid-owner 989 -j DNAT --to-destination 10.100.1.224:9319
ERROR: passthru not allowed: iptables
[ERR]  Could not add iptables redirect (iptables not allowed in wrapper)
[ERR]  ╔════════════════════════════════════════════════════════════════════════╗
[ERR]  ║  CRITICAL: Tempo search will NOT work without this redirect!           ║
[ERR]  ║                                                                        ║
[ERR]  ║  Root cause: Tempo single-binary dials 127.0.0.1:<grpc_port>          ║
[ERR]  ║  Each instance needs unique port + matching DNAT rule.                ║
[ERR]  ║                                                                        ║
[ERR]  ║  Manually run:                                                         ║
[ERR]  ║  sudo iptables -t nat -A OUTPUT -p tcp -d 127.0.0.1 --dport 9319 -m owner --uid-owner 989 \                   ║
[ERR]  ║    -j DNAT --to-destination 10.100.1.224:9319                                                  ║
[ERR]  ╚════════════════════════════════════════════════════════════════════════╝
[INFO] Enabling and starting Grafana Tempo service...
[2026-01-12 20:37:04 UTC] USER=www-data EUID=0 PID=690510 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-12 20:37:05 UTC] USER=www-data EUID=0 PID=690556 ACTION=passthru ARGS=systemctl enable tempo-obs-user-universe-main-dev.service
[2026-01-12 20:37:05 UTC] USER=www-data EUID=0 PID=690622 ACTION=passthru ARGS=systemctl restart tempo-obs-user-universe-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-01-12 20:37:13 UTC] USER=www-data EUID=0 PID=690903 ACTION=passthru ARGS=systemctl is-active --quiet tempo-obs-user-universe-main-dev.service
[OK]   ✅ Grafana Tempo is running
[OK]   ✅ HTTP endpoint listening on port 3200
[OK]   ✅ OTLP gRPC endpoint listening on port 4317
[OK]   ✅ OTLP HTTP endpoint listening on port 4318
[INFO] Running smoke test: Tempo search endpoint...
[WARN] ⚠️  Tempo search smoke test failed - check iptables DNAT rule
[WARN]     Expected JSON with completedJobs/totalJobs, got: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Unavailable</titl
[INFO] Service logs (last 10 lines):
[2026-01-12 20:37:16 UTC] USER=www-data EUID=0 PID=691007 ACTION=passthru ARGS=journalctl -u tempo-obs-user-universe-main-dev.service -n 10 --no-pager
Jan 12 20:37:11 web-03 tempo-obs-user-universe-main-dev[690792]: level=info ts=2026-01-12T20:37:11.183555687Z caller=lifecycler.go:687 msg="not loading tokens from file, tokens file path is empty"
Jan 12 20:37:11 web-03 tempo-obs-user-universe-main-dev[690792]: ts=2026-01-12T20:37:11Z level=info msg="Starting gRPC server for Jaeger Protobuf" component=tempo endpoint=10.100.1.224:14474
Jan 12 20:37:11 web-03 tempo-obs-user-universe-main-dev[690792]: level=info ts=2026-01-12T20:37:11.183880966Z caller=lifecycler.go:714 msg="instance not found in ring, adding with no tokens" ring=ingester
Jan 12 20:37:11 web-03 tempo-obs-user-universe-main-dev[690792]: level=info ts=2026-01-12T20:37:11.183953164Z caller=lifecycler.go:556 msg="auto-joining cluster after timeout" ring=ingester
Jan 12 20:37:11 web-03 tempo-obs-user-universe-main-dev[690792]: level=info ts=2026-01-12T20:37:11.183839407Z caller=compactor.go:125 msg="waiting until compactor is ACTIVE in the ring"
Jan 12 20:37:11 web-03 tempo-obs-user-universe-main-dev[690792]: level=info ts=2026-01-12T20:37:11.18433485Z caller=compactor.go:131 msg="compactor is ACTIVE in the ring"
Jan 12 20:37:11 web-03 tempo-obs-user-universe-main-dev[690792]: level=info ts=2026-01-12T20:37:11.184348176Z caller=compactor.go:140 msg="waiting until compactor ring topology is stable" min_waiting=1m0s max_waiting=5m0s
Jan 12 20:37:11 web-03 tempo-obs-user-universe-main-dev[690792]: level=info ts=2026-01-12T20:37:11.184895087Z caller=worker.go:250 msg="total worker concurrency updated" totalConcurrency=20
Jan 12 20:37:11 web-03 tempo-obs-user-universe-main-dev[690792]: ts=2026-01-12T20:37:11Z level=info msg="Starting GRPC server" component=tempo endpoint=10.100.1.224:4317
Jan 12 20:37:11 web-03 tempo-obs-user-universe-main-dev[690792]: ts=2026-01-12T20:37:11Z level=info msg="Starting HTTP server" component=tempo endpoint=10.100.1.224:4318

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Traces Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: tempo
[INFO] FQDN: traces-user-universe-main-dev-tempo.fastorder.com
[INFO] IP: 10.100.1.224
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering Tempo in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Tempo
[INFO]   Identifier:        user-universe-main-dev-tempo
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.224
[INFO]   Port:              3200
[INFO]   FQDN:              traces-user-universe-main-dev-tempo.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-universe-main-dev (service=user, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 4d614318-3111-457b-a8b1-acfec8ba5be5
[SUCCESS] Environment UUID: b683e8be-3b47-4d33-9d5c-389c63a0ae74
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b683e8be-3b47-4d33-9d5c-389c63a0ae74
[OK]   ✅ Tempo registered in dashboard
[OK]   Traces backend (tempo) deployed successfully
[INFO] Step 9/10: Deploying dashboards...
[INFO]   Provider: grafana (selected)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] DASHBOARDS DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: grafana
[INFO] Observability Cell: obs-user-universe-main-dev
[INFO] FQDN: dashboards-user-universe-main-dev-grafana.fastorder.com
[INFO] IP: 10.100.1.222
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: grafana
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Dashboards/provider/grafana.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=user, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Binding to allocated IP: 10.100.1.222
[INFO] Deploying Grafana for observability cell: obs-user-universe-main-dev
[INFO] FQDN: dashboards-user-universe-main-dev-grafana.fastorder.com
[INFO] Allocated IP: 10.100.1.222
[INFO] VM IP: 10.100.1.222
[INFO] HTTP Port: 3000
[INFO] Checking if Grafana is installed...
[OK]   Grafana already installed
[INFO] Installing Grafana plugins...
[INFO] Installing ClickHouse datasource plugin...
[WARN] Failed to install ClickHouse plugin (may need internet access)
[INFO] Validating TLS certificate and key...
[INFO] Setting certificate permissions...
[OK]   TLS cert/key found and permissions set
[INFO] Creating configuration and data directories...
[2026-01-12 20:37:16 UTC] USER=www-data EUID=0 PID=691121 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-user-universe-main-dev
[2026-01-12 20:37:16 UTC] USER=www-data EUID=0 PID=691130 ACTION=passthru ARGS=mkdir -p /var/lib/grafana/obs-user-universe-main-dev
[2026-01-12 20:37:16 UTC] USER=www-data EUID=0 PID=691139 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-user-universe-main-dev/provisioning/datasources
[2026-01-12 20:37:16 UTC] USER=www-data EUID=0 PID=691150 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-user-universe-main-dev/provisioning/dashboards
[INFO] Creating Grafana configuration at /etc/grafana/obs-user-universe-main-dev/grafana.ini...
[OK]   Configuration created
[INFO] Creating Prometheus datasource provisioning...
[OK]   Prometheus datasource provisioned
[INFO] Creating Tempo datasource provisioning...
[OK]   Tempo datasource provisioned
[INFO] Creating Loki datasource provisioning...
[OK]   Loki datasource provisioned
[INFO] Creating ClickHouse datasource provisioning...
[OK]   Retrieved ClickHouse credentials from Secrets Manager
[OK]   ClickHouse datasource provisioned
[INFO] Creating systemd service: grafana-obs-user-universe-main-dev
[OK]   Systemd service created
[2026-01-12 20:37:18 UTC] USER=www-data EUID=0 PID=691256 ACTION=passthru ARGS=chown -R grafana:grafana /etc/grafana/obs-user-universe-main-dev
[2026-01-12 20:37:18 UTC] USER=www-data EUID=0 PID=691265 ACTION=passthru ARGS=chown -R grafana:grafana /var/lib/grafana/obs-user-universe-main-dev
[2026-01-12 20:37:18 UTC] USER=www-data EUID=0 PID=691274 ACTION=passthru ARGS=chmod 750 /etc/grafana/obs-user-universe-main-dev /var/lib/grafana/obs-user-universe-main-dev
[INFO] Adding /etc/hosts entry for dashboards-user-universe-main-dev-grafana.fastorder.com -> 10.100.1.222
[WARN] /etc/hosts entry already exists
[INFO] Storing Grafana credentials in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/user/universe/main/dev/grafana/admin-EVtjlN",
    "Name": "fastorder/observability/user/universe/main/dev/grafana/admin",
    "VersionId": "3932121d-0a2b-44b0-9cd1-54ce0e2b4359"
}
[OK]   Credentials stored in AWS Secrets Manager: fastorder/observability/user/universe/main/dev/grafana/admin
[INFO] Enabling and starting Grafana service...
[2026-01-12 20:37:21 UTC] USER=www-data EUID=0 PID=691308 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-12 20:37:21 UTC] USER=www-data EUID=0 PID=691356 ACTION=passthru ARGS=systemctl enable grafana-obs-user-universe-main-dev.service
[2026-01-12 20:37:22 UTC] USER=www-data EUID=0 PID=691411 ACTION=passthru ARGS=systemctl restart grafana-obs-user-universe-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-01-12 20:37:27 UTC] USER=www-data EUID=0 PID=691528 ACTION=passthru ARGS=systemctl is-active --quiet grafana-obs-user-universe-main-dev.service
[OK]   ✅ Grafana is running
[OK]   ✅ Grafana web interface listening on port 3000
[INFO] ═══════════════════════════════════════════════════════════════
[OK]   Grafana Dashboard URL: https://dashboards-user-universe-main-dev-grafana.fastorder.com:3000
[OK]   Username: admin
[OK]   Password is stored in AWS Secrets Manager at: fastorder/observability/user/universe/main/dev/grafana/admin
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service logs (last 10 lines):
[2026-01-12 20:37:27 UTC] USER=www-data EUID=0 PID=691539 ACTION=passthru ARGS=journalctl -u grafana-obs-user-universe-main-dev.service -n 10 --no-pager

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Dashboards Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: grafana
[INFO] FQDN: dashboards-user-universe-main-dev-grafana.fastorder.com
[INFO] IP: 10.100.1.222
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering Grafana in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Grafana
[INFO]   Identifier:        user-universe-main-dev-grafana
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.222
[INFO]   Port:              3000
[INFO]   FQDN:              dashboards-user-universe-main-dev-grafana.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-universe-main-dev (service=user, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: b70562c9-1548-47bb-8b55-e0be864bb7a9
[SUCCESS] Environment UUID: b683e8be-3b47-4d33-9d5c-389c63a0ae74
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b683e8be-3b47-4d33-9d5c-389c63a0ae74
[OK]   ✅ Grafana registered in dashboard
[OK]   Dashboards (grafana) deployed successfully
[INFO] Step 10/10: Deploying alerting...
[INFO]   Provider: alertmanager (selected)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] ALERTING DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: alertmanager
[INFO] Observability Cell: obs-user-universe-main-dev
[INFO] FQDN: alerts-user-universe-main-dev-alertmanager.fastorder.com
[INFO] IP: 10.100.1.225
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Ports: Web=9093 Cluster=9094 (bound to IP: 10.100.1.225)

[INFO] Using provider: alertmanager
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Alerting/provider/alertmanager.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=user, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Binding to allocated IP: 10.100.1.225
[INFO] Deploying Alertmanager for observability cell: obs-user-universe-main-dev
[INFO] FQDN: alerts-user-universe-main-dev-alertmanager.fastorder.com
[INFO] Allocated IP: 10.100.1.225
[INFO] VM IP: 10.100.1.225
[INFO] Ports: Web=9093 Cluster=9094
[INFO] Checking if Alertmanager is installed...
[OK]   Alertmanager already installed at /usr/local/bin/alertmanager
[INFO] Validating TLS certificate and key...
[OK]   TLS cert/key found in /etc/fastorder/observability/certs/obs-user-universe-main-dev
[INFO] Creating configuration and data directories...
[2026-01-12 20:37:28 UTC] USER=www-data EUID=0 PID=691600 ACTION=passthru ARGS=mkdir -p /etc/alertmanager/obs-user-universe-main-dev
[2026-01-12 20:37:28 UTC] USER=www-data EUID=0 PID=691615 ACTION=passthru ARGS=mkdir -p /var/lib/alertmanager/obs-user-universe-main-dev
[2026-01-12 20:37:28 UTC] USER=www-data EUID=0 PID=691625 ACTION=passthru ARGS=mkdir -p /etc/alertmanager/obs-user-universe-main-dev/templates
[INFO] Creating Alertmanager configuration...
[OK]   Alertmanager configuration created at /etc/alertmanager/obs-user-universe-main-dev/alertmanager.yml
[INFO] Creating notification templates...
[OK]   Notification templates created
[INFO] Creating Alertmanager web TLS configuration with mTLS...
[OK]   Web mTLS configuration created at /etc/alertmanager/obs-user-universe-main-dev/web-config.yml
[INFO] Validating Alertmanager configuration...
[2026-01-12 20:37:28 UTC] USER=www-data EUID=0 PID=691661 ACTION=passthru ARGS=chmod 755 /etc/alertmanager/obs-user-universe-main-dev
[2026-01-12 20:37:28 UTC] USER=www-data EUID=0 PID=691670 ACTION=passthru ARGS=chmod 644 /etc/alertmanager/obs-user-universe-main-dev/alertmanager.yml
Checking '/etc/alertmanager/obs-user-universe-main-dev/alertmanager.yml'  SUCCESS
Found:
 - global config
 - route
 - 6 inhibit rules
 - 5 receivers
 - 1 templates
  SUCCESS

[OK]   ✅ Configuration is valid
[INFO] Creating systemd service: alertmanager-obs-user-universe-main-dev
[OK]   Systemd service created
[2026-01-12 20:37:28 UTC] USER=www-data EUID=0 PID=691700 ACTION=passthru ARGS=chown alertmanager:alertmanager /etc/fastorder/observability/certs/obs-user-universe-main-dev/alertmanager-key.pem
[2026-01-12 20:37:28 UTC] USER=www-data EUID=0 PID=691717 ACTION=passthru ARGS=chown alertmanager:alertmanager /etc/fastorder/observability/certs/obs-user-universe-main-dev/alertmanager-cert.pem
[2026-01-12 20:37:28 UTC] USER=www-data EUID=0 PID=691735 ACTION=passthru ARGS=chown -R alertmanager:alertmanager /etc/alertmanager/obs-user-universe-main-dev
[2026-01-12 20:37:28 UTC] USER=www-data EUID=0 PID=691744 ACTION=passthru ARGS=chown -R alertmanager:alertmanager /var/lib/alertmanager/obs-user-universe-main-dev
[2026-01-12 20:37:28 UTC] USER=www-data EUID=0 PID=691753 ACTION=passthru ARGS=chmod 750 /etc/alertmanager/obs-user-universe-main-dev /var/lib/alertmanager/obs-user-universe-main-dev
[INFO] Adding /etc/hosts entry for alerts-user-universe-main-dev-alertmanager.fastorder.com -> 10.100.1.225
[WARN] /etc/hosts entry already exists
[INFO] Storing Alertmanager configuration in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/user/universe/main/dev/alertmanager/server-4xaPck",
    "Name": "fastorder/observability/user/universe/main/dev/alertmanager/server",
    "VersionId": "f06548a2-4cff-4d48-bc0e-5c2cbc2499de"
}
[OK]   Configuration stored in AWS Secrets Manager: fastorder/observability/user/universe/main/dev/alertmanager/server
[INFO] Enabling and starting Alertmanager service...
[2026-01-12 20:37:31 UTC] USER=www-data EUID=0 PID=691795 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-12 20:37:32 UTC] USER=www-data EUID=0 PID=691855 ACTION=passthru ARGS=systemctl enable alertmanager-obs-user-universe-main-dev.service
[2026-01-12 20:37:32 UTC] USER=www-data EUID=0 PID=691914 ACTION=passthru ARGS=systemctl restart alertmanager-obs-user-universe-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-01-12 20:37:35 UTC] USER=www-data EUID=0 PID=691963 ACTION=passthru ARGS=systemctl is-active --quiet alertmanager-obs-user-universe-main-dev.service
[OK]   ✅ Alertmanager is running
[OK]   ✅ Alertmanager HTTPS web interface listening on port 9093
[OK]   ✅ Alertmanager cluster port listening on port 9094
[WARN] ⚠️  Alertmanager health check not responding yet (HTTPS)
[INFO] ═══════════════════════════════════════════════════════════════
[OK]   Alertmanager Web UI: https://alerts-user-universe-main-dev-alertmanager.fastorder.com:9093
[OK]   API Endpoint:        https://alerts-user-universe-main-dev-alertmanager.fastorder.com:9093/api/v2
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service logs (last 10 lines):
[2026-01-12 20:37:35 UTC] USER=www-data EUID=0 PID=691977 ACTION=passthru ARGS=journalctl -u alertmanager-obs-user-universe-main-dev.service -n 10 --no-pager
Jan 12 20:37:32 web-03 systemd[1]: Started Alertmanager - obs-user-universe-main-dev.
Jan 12 20:37:33 web-03 alertmanager-obs-user-universe-main-dev[691924]: ts=2026-01-12T20:37:33.033Z caller=main.go:245 level=info msg="Starting Alertmanager" version="(version=0.26.0, branch=HEAD, revision=d7b4f0c7322e7151d6e3b1e31cbc15361e295d8d)"
Jan 12 20:37:33 web-03 alertmanager-obs-user-universe-main-dev[691924]: ts=2026-01-12T20:37:33.034Z caller=main.go:246 level=info build_context="(go=go1.20.7, platform=linux/amd64, user=root@df8d7debeef4, date=20230824-11:11:58, tags=netgo)"
Jan 12 20:37:33 web-03 alertmanager-obs-user-universe-main-dev[691924]: ts=2026-01-12T20:37:33.037Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s
Jan 12 20:37:33 web-03 alertmanager-obs-user-universe-main-dev[691924]: ts=2026-01-12T20:37:33.089Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/obs-user-universe-main-dev/alertmanager.yml
Jan 12 20:37:33 web-03 alertmanager-obs-user-universe-main-dev[691924]: ts=2026-01-12T20:37:33.091Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/obs-user-universe-main-dev/alertmanager.yml
Jan 12 20:37:33 web-03 alertmanager-obs-user-universe-main-dev[691924]: ts=2026-01-12T20:37:33.095Z caller=tls_config.go:274 level=info msg="Listening on" address=10.100.1.225:9093
Jan 12 20:37:33 web-03 alertmanager-obs-user-universe-main-dev[691924]: ts=2026-01-12T20:37:33.095Z caller=tls_config.go:310 level=info msg="TLS is enabled." http2=true address=10.100.1.225:9093
Jan 12 20:37:35 web-03 alertmanager-obs-user-universe-main-dev[691924]: ts=2026-01-12T20:37:35.039Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000019084s
Jan 12 20:37:35 web-03 alertmanager-obs-user-universe-main-dev[691924]: 2026/01/12 20:37:35 http: TLS handshake error from 10.100.1.225:54960: tls: client didn't provide a certificate

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Alerting Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: alertmanager
[INFO] FQDN: alerts-user-universe-main-dev-alertmanager.fastorder.com
[INFO] IP: 10.100.1.225
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Setting up HTTPS reverse proxy...
[INFO] Backend port: 9093
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Alertmanager HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-user-universe-main-dev
  FQDN:         alerts-user-universe-main-dev-alertmanager.fastorder.com
  Backend:      https://alerts-user-universe-main-dev-alertmanager.fastorder.com:9093/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.225
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[ERROR] This script must be run as root or with sudo
[WARN] ⚠️  HTTPS setup failed (Alertmanager is still running on HTTP)
[INFO] Registering Alertmanager in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Alertmanager
[INFO]   Identifier:        user-universe-main-dev-alertmanager
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.225
[INFO]   Port:              9093
[INFO]   FQDN:              alerts-user-universe-main-dev-alertmanager.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-universe-main-dev (service=user, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 755e4db0-cda5-41ac-bdb0-f41cd95f82af
[SUCCESS] Environment UUID: b683e8be-3b47-4d33-9d5c-389c63a0ae74
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b683e8be-3b47-4d33-9d5c-389c63a0ae74
[OK]   ✅ Alertmanager registered in dashboard
[OK]   Alerting (alertmanager) deployed successfully
[INFO] Step 10.5: Deploying Blackbox Exporter for synthetic monitoring...
[BLACKBOX] Starting Blackbox Exporter deployment for obs-user-universe-main-dev
[BLACKBOX] VM IP: 10.100.1.221
[BLACKBOX] Version: 0.25.0
[BLACKBOX] Checking prerequisites...
[BLACKBOX] Creating directories...
[BLACKBOX] Downloading Blackbox Exporter v0.25.0...
Sorry, user www-data is not allowed to execute '/usr/bin/mv /tmp/tmp.bcgetIFcFh/blackbox_exporter-0.25.0.linux-amd64/blackbox_exporter /usr/local/bin/' as root on web-03.
[WARN] Blackbox Exporter deployment failed (non-fatal, synthetic monitoring disabled)
[INFO] Step 11/13: Configuring HTTPS reverse proxies...
[INFO] Setting up Prometheus HTTPS proxy...
[2026-01-12 20:37:37 UTC] USER=www-data EUID=0 PID=692110 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/https/setup-prometheus-https.sh --obs-cell obs-user-universe-main-dev --backend-ip 10.100.1.221
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Prometheus HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-user-universe-main-dev
  FQDN:         metrics-user-universe-main-dev-prometheus.fastorder.com
  Backend:      https://metrics-user-universe-main-dev-prometheus.fastorder.com:9090/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.221
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity (will retry up to 60s)...
[OK] Backend is accessible
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate exists and is valid for 88 more days
[INFO] Creating HTTPS VirtualHost with reverse proxy...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Setting up certificate auto-renewal...
[OK] Auto-renewal configured
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated
[INFO] Verifying HTTPS setup...
[OK] HTTPS endpoint is working

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Prometheus HTTPS Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HTTPS Endpoints:
  Health:   https://metrics-user-universe-main-dev-prometheus.fastorder.com/-/healthy
  Ready:    https://metrics-user-universe-main-dev-prometheus.fastorder.com/-/ready
  Graph:    https://metrics-user-universe-main-dev-prometheus.fastorder.com/graph
  Targets:  https://metrics-user-universe-main-dev-prometheus.fastorder.com/targets
  Alerts:   https://metrics-user-universe-main-dev-prometheus.fastorder.com/alerts
  API:      https://metrics-user-universe-main-dev-prometheus.fastorder.com/api/v1/...

Apache VirtualHosts:
  HTTP:  /etc/apache2/sites-available/metrics-user-universe-main-dev-prometheus.fastorder.com.conf
  HTTPS: /etc/apache2/sites-available/metrics-user-universe-main-dev-prometheus.fastorder.com-ssl.conf

Certificate:
  Path:     /etc/letsencrypt/live/metrics-user-universe-main-dev-prometheus.fastorder.com/
  Renewal:  certbot renew --cert-name metrics-user-universe-main-dev-prometheus.fastorder.com

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   Prometheus HTTPS proxy configured
[INFO] Setting up Grafana HTTPS proxy...
[2026-01-12 20:37:40 UTC] USER=www-data EUID=0 PID=692274 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Dashboards/https/setup-grafana-https.sh --obs-cell obs-user-universe-main-dev --backend-ip 10.100.1.222
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-user-universe-main-dev
  FQDN:         dashboards-user-universe-main-dev-grafana.fastorder.com
  Backend:      https://dashboards-user-universe-main-dev-grafana.fastorder.com:3000/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.222
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate already exists
[INFO] Creating HTTPS VirtualHost...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana HTTPS Setup Complete!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  Grafana URL: https://dashboards-user-universe-main-dev-grafana.fastorder.com/
  Metrics:     https://dashboards-user-universe-main-dev-grafana.fastorder.com/metrics

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   Grafana HTTPS proxy configured
[INFO] Setting up OpenTelemetry Collector HTTPS proxy...
[2026-01-12 20:37:40 UTC] USER=www-data EUID=0 PID=692341 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Telemetry/https/setup-otelcol-https.sh --obs-cell obs-user-universe-main-dev --backend-ip 10.100.1.226
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OpenTelemetry Collector HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-user-universe-main-dev
  FQDN:         telemetry-user-universe-main-dev-opentelemetry.fastorder.com
  Backend:      http://telemetry-user-universe-main-dev-opentelemetry.fastorder.com:8888/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.226
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[OK] Backend is accessible and returning metrics via HTTPS
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate exists and is valid for 88 more days
[INFO] Creating HTTPS VirtualHost with reverse proxy...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Setting up certificate auto-renewal...
[OK] Auto-renewal configured
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated
[INFO] Verifying HTTPS setup...
[OK] HTTPS endpoint is working and returning metrics

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OpenTelemetry Collector HTTPS Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HTTPS Endpoints:
  Metrics: https://telemetry-user-universe-main-dev-opentelemetry.fastorder.com/metrics

Apache VirtualHosts:
  HTTP:  /etc/apache2/sites-available/telemetry-user-universe-main-dev-opentelemetry.fastorder.com.conf
  HTTPS: /etc/apache2/sites-available/telemetry-user-universe-main-dev-opentelemetry.fastorder.com-ssl.conf

Certificate:
  Path:     /etc/letsencrypt/live/telemetry-user-universe-main-dev-opentelemetry.fastorder.com/
  Renewal:  certbot renew --cert-name telemetry-user-universe-main-dev-opentelemetry.fastorder.com

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   OpenTelemetry Collector HTTPS proxy configured
[INFO] Setting up ClickHouse HTTPS proxy...
[2026-01-12 20:37:43 UTC] USER=www-data EUID=0 PID=692517 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/https/setup-clickhouse-https.sh --obs-cell obs-user-universe-main-dev --backend-ip 10.100.1.223
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ClickHouse HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-user-universe-main-dev
  FQDN:         logstore-user-universe-main-dev.fastorder.com
  Backend:      http://logstore-user-universe-main-dev.fastorder.com:8123/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.223
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity (will retry up to 60s)...
[OK] Backend is accessible
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate exists and is valid for 88 more days
[INFO] Creating HTTPS VirtualHost with reverse proxy...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Setting up certificate auto-renewal...
[OK] Auto-renewal configured
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated
[INFO] Verifying HTTPS setup...
[OK] HTTPS endpoint is working

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ ClickHouse HTTPS Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HTTPS Endpoints:
  Health:     https://logstore-user-universe-main-dev.fastorder.com/
  Dashboard:  https://logstore-user-universe-main-dev.fastorder.com/dashboard
  Playground: https://logstore-user-universe-main-dev.fastorder.com/play
  Metrics:    https://logstore-user-universe-main-dev.fastorder.com/metrics

Login Instructions:
  1. Get credentials from skeleton: POST /api/monitoring/clickhouse/credentials
  2. Use auto-login URL: https://logstore-user-universe-main-dev.fastorder.com/dashboard#user=<USER>&password=<PASS>
  3. Or use skeleton monitoring dashboard for one-click access

Apache VirtualHosts:
  HTTP:  /etc/apache2/sites-available/logstore-user-universe-main-dev.fastorder.com.conf
  HTTPS: /etc/apache2/sites-available/logstore-user-universe-main-dev.fastorder.com-ssl.conf

Certificate:
  Path: /etc/letsencrypt/live/logstore-user-universe-main-dev.fastorder.com/
  Auto-renewal: Enabled via certbot.timer

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ClickHouse HTTPS proxy configured
[INFO] Setting up Tempo HTTPS proxy...
[2026-01-12 20:37:46 UTC] USER=www-data EUID=0 PID=692666 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Traces/https/setup-tempo-https.sh --obs-cell obs-user-universe-main-dev --backend-ip 10.100.1.224
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana Tempo HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-user-universe-main-dev
  FQDN:         traces-user-universe-main-dev-tempo.fastorder.com
  Backend:      https://10.100.1.224:3200/
  Backend IP:   10.100.1.224
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[WARN] Cannot verify Tempo health endpoint (it may not be running yet), continuing anyway...
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate already exists
[INFO] Generating Apache client certificate for mTLS backend connection...
[OK] Apache client certificate already exists
[INFO] Creating HTTPS VirtualHost with mTLS backend...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana Tempo HTTPS Setup Complete!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  Tempo URL:    https://traces-user-universe-main-dev-tempo.fastorder.com/
  Ready:        https://traces-user-universe-main-dev-tempo.fastorder.com/ready
  Metrics:      https://traces-user-universe-main-dev-tempo.fastorder.com/metrics
  Build Info:   https://traces-user-universe-main-dev-tempo.fastorder.com/api/status/buildinfo

  Note: Tempo backend must be running at traces-user-universe-main-dev-tempo.fastorder.com:3200 (10.100.1.224)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   Tempo HTTPS proxy configured
[INFO] Setting up Alertmanager HTTPS proxy...
[2026-01-12 20:37:47 UTC] USER=www-data EUID=0 PID=692772 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Alerting/https/setup-alertmanager-https.sh --obs-cell obs-user-universe-main-dev --backend-ip 10.100.1.225
[INFO] Backend port: 9093
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Alertmanager HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-user-universe-main-dev
  FQDN:         alerts-user-universe-main-dev-alertmanager.fastorder.com
  Backend:      https://alerts-user-universe-main-dev-alertmanager.fastorder.com:9093/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.225
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[WARN] Backend health check inconclusive - proceeding anyway
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate exists and is valid for 88 more days
[INFO] Creating HTTPS VirtualHost with reverse proxy...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Setting up certificate auto-renewal...
[OK] Auto-renewal configured
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated
[INFO] Verifying HTTPS setup...
OK[OK] HTTPS endpoint is working

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Alertmanager HTTPS Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HTTPS Endpoints:
  Health:   https://alerts-user-universe-main-dev-alertmanager.fastorder.com/-/healthy
  Ready:    https://alerts-user-universe-main-dev-alertmanager.fastorder.com/-/ready
  Web UI:   https://alerts-user-universe-main-dev-alertmanager.fastorder.com/
  API:      https://alerts-user-universe-main-dev-alertmanager.fastorder.com/api/v2/...

Apache VirtualHosts:
  HTTP:  /etc/apache2/sites-available/alerts-user-universe-main-dev-alertmanager.fastorder.com.conf
  HTTPS: /etc/apache2/sites-available/alerts-user-universe-main-dev-alertmanager.fastorder.com-ssl.conf

Certificate:
  Path:     /etc/letsencrypt/live/alerts-user-universe-main-dev-alertmanager.fastorder.com/
  Renewal:  certbot renew --cert-name alerts-user-universe-main-dev-alertmanager.fastorder.com

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   Alertmanager HTTPS proxy configured
[OK]   HTTPS reverse proxies configured
[INFO] Step 12/13: Configuring firewall rules (network segmentation)...

[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING FIREWALL RULES FOR OBSERVABILITY CELL
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Cell ID: obs-user-universe-main-dev
[INFO] Internal Network: 10.0.0.0/8

[INFO] Discovering dashboard/skeleton VM IPs...
[INFO]   Discovered skeleton IP: 142.93.238.16 (skeleton.fastorder.com)
[INFO] Authorized dashboard IPs:
[INFO]   - 10.100.60.2
[INFO]   - 142.93.238.16

[INFO] Configuring UFW firewall rules...
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=692986 ACTION=passthru ARGS=ufw --force enable
ERROR: passthru not allowed: ufw
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=692994 ACTION=passthru ARGS=ufw default deny incoming
ERROR: passthru not allowed: ufw
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=693010 ACTION=passthru ARGS=ufw allow 22/tcp comment SSH
ERROR: passthru not allowed: ufw
[INFO]   Allowing prometheus (port 9090) from internal network...
[INFO]   Allowing alertmanager (port 9093) from internal network...
[INFO]   Allowing clickhouse (port 8123) from internal network...
[INFO]   Allowing grafana (port 3000) from internal network...
[INFO]   Allowing otelcol (port 4318) from internal network...
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=693050 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 4318 proto tcp comment Obs: otelcol from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing loki (port 3100) from internal network...
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=693058 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 3100 proto tcp comment Obs: loki from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing tempo (port 3200) from internal network...
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=693066 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 3200 proto tcp comment Obs: tempo from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing dashboard access from 10.100.60.2...
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=693074 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 9090 proto tcp comment Dashboard: prometheus
ERROR: passthru not allowed: ufw
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=693114 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3100 proto tcp comment Dashboard: loki
ERROR: passthru not allowed: ufw
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=693122 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3200 proto tcp comment Dashboard: tempo
ERROR: passthru not allowed: ufw
[INFO]   Allowing dashboard access from 142.93.238.16...
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=693130 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 9090 proto tcp comment Dashboard: prometheus
ERROR: passthru not allowed: ufw
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=693138 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 9093 proto tcp comment Dashboard: alertmanager
ERROR: passthru not allowed: ufw
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=693146 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 8123 proto tcp comment Dashboard: clickhouse
ERROR: passthru not allowed: ufw
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=693154 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 3000 proto tcp comment Dashboard: grafana
ERROR: passthru not allowed: ufw
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=693162 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 4318 proto tcp comment Dashboard: otelcol
ERROR: passthru not allowed: ufw
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=693170 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 3100 proto tcp comment Dashboard: loki
ERROR: passthru not allowed: ufw
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=693180 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 3200 proto tcp comment Dashboard: tempo
ERROR: passthru not allowed: ufw
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=693194 ACTION=passthru ARGS=ufw allow 443/tcp comment HTTPS obs-proxy
ERROR: passthru not allowed: ufw
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=693202 ACTION=passthru ARGS=ufw reload
ERROR: passthru not allowed: ufw
[OK]   UFW firewall rules configured

[OK]   ═══════════════════════════════════════════════════════════════
[OK]   ✅ Firewall configuration completed
[OK]   ═══════════════════════════════════════════════════════════════

[INFO] Current firewall status:
[2026-01-12 20:37:50 UTC] USER=www-data EUID=0 PID=693221 ACTION=passthru ARGS=ufw status numbered
ERROR: passthru not allowed: ufw
[OK]   Firewall rules configured
[INFO] Step 13/13: Configuring OAuth/SSO...
[INFO] OAuth/SSO configuration script not found, skipping...

[INFO] Running validation checks...
[INFO] Validation script not found, skipping...

[INFO] Registering observability components to dashboard...
[INFO] Components to register: metrics alerts dashboards traces telemetry logstore proxy
[INFO]   Skipping metrics - registered by deploy script
[INFO]   Skipping alerts - registered by deploy script
[INFO]   Skipping dashboards - registered by deploy script
[INFO]   Skipping traces - registered by deploy script
[INFO]   Skipping telemetry - registered by deploy script
[INFO]   Skipping logstore - registered by deploy script
[INFO]   Processing component: proxy
[INFO] Registering: proxy (obs-user-universe-main-dev-proxy)
[INFO] Detected observability component, parsing: user-universe-main-dev-proxy
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Observability Proxy
[INFO]   Identifier:        obs-user-universe-main-dev-proxy
[INFO]   Identifier Parent: observability-cell
[INFO]   IP:                10.100.1.220
[INFO]   Port:              443
[INFO]   FQDN:              observe-user-universe-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-universe-main-dev (service=user, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 1b780fee-ca67-4ebd-832c-19ca90c69130
[SUCCESS] Environment UUID: b683e8be-3b47-4d33-9d5c-389c63a0ae74
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b683e8be-3b47-4d33-9d5c-389c63a0ae74
[OK]   ✓ Registered: proxy
[INFO] Registering short DNS aliases...
[OK]   ✓ Observability components registration completed

[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] Verifying all observability services are running...
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[OK]     ✓ prometheus-obs-user-universe-main-dev.service is running
[OK]     ✓ alertmanager-obs-user-universe-main-dev.service is running
[OK]     ✓ clickhouse-server-obs-user-universe-main-dev.service is running
[OK]   ✓ All observability services verified running


═══════════════════════════════════════════════════════════════════════════════
[OK]   ✅ OBSERVABILITY CELL PROVISIONED: obs-user-universe-main-dev
═══════════════════════════════════════════════════════════════════════════════

[INFO] DNS Entries:
  metrics-user-universe-main-dev-prometheus.fastorder.com (10.100.1.221)
  alerts-user-universe-main-dev-alertmanager.fastorder.com (10.100.1.225)
  dashboards-user-universe-main-dev-grafana.fastorder.com (10.100.1.222)
  traces-user-universe-main-dev-tempo.fastorder.com (10.100.1.224)
  telemetry-user-universe-main-dev-opentelemetry.fastorder.com (10.100.1.226)
  logstore-user-universe-main-dev-clickhouse.fastorder.com (10.100.1.223)
  observe-user-universe-main-dev.fastorder.com (10.100.1.220)

[INFO] Secrets Path: fastorder/observability/user/universe/dev/*

[INFO] Access (Purpose-Oriented URLs):
  Dashboards: https://dashboards-user-universe-main-dev-grafana.fastorder.com (SSO enabled)
  Metrics: https://metrics-user-universe-main-dev-prometheus.fastorder.com (internal only)
  Alerts: https://alerts-user-universe-main-dev-alertmanager.fastorder.com
  Log Storage: https://logstore-user-universe-main-dev-clickhouse.fastorder.com

[INFO] Backend Implementation (Internal - Not Exposed to Clients):
  Telemetry: otlp
  Metrics: prometheus
  Traces: tempo
  Dashboards: grafana
  Alerting: alertmanager
  Log Storage: clickhouse

[INFO] For applications in user-universe-main-dev:
  - Metrics: Push to telemetry-user-universe-main-dev-opentelemetry.fastorder.com:4318 (OTLP/HTTP)
  - Logs: Push to telemetry-user-universe-main-dev-opentelemetry.fastorder.com:4318 (OTLP/HTTP)
  - Traces: Push to telemetry-user-universe-main-dev-opentelemetry.fastorder.com:4317 (OTLP/gRPC)
  - Query Metrics: https://metrics-user-universe-main-dev-prometheus.fastorder.com
  - Query Logs: https://logstore-user-universe-main-dev-clickhouse.fastorder.com
  - Query Traces: https://traces-user-universe-main-dev-tempo.fastorder.com

[INFO] Runbook: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/RUNBOOK.md
═══════════════════════════════════════════════════════════════════════════════

[INFO] Using search engine from SEARCH_ENGINE environment variable: elasticsearch
[INFO] Cleaning up any existing locks...

Starting search engine: elasticsearch
═══════════════════════════════════════════════

════════════════════════════════════════════════════════════════
           Elasticsearch Deployment Runner                        
════════════════════════════════════════════════════════════════

[INFO] Cleaning up any existing locks (without triggering package configurations)...
[WARNING] Lock cleanup skipped (wrapper not available or insufficient permissions)

🚀 Auto mode enabled - running automatic installation


Starting Automatic Installation...
═══════════════════════════════════════════════
Will execute all deployment tasks in sequence:

  [1] Install Elasticsearch Http (01-install-elasticsearch-http)
  [2] Make Https (02-make-https)
  [3] Create Index Llm (03-create-index-llm)
  [4] Monitoring Setup (10-monitoring-setup)

═══════════════════════════════════════════════
🚀 Auto mode - proceeding automatically...

Running automatic installation...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 1: Executing Install Elasticsearch Http
Folder: 01-install-elasticsearch-http
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

=== Elasticsearch HTTP Setup ===
Install and configure Elasticsearch with HTTP access
Architecture: Per-node VM IPs with default port (9200)

[INFO] Using web-provided environment: user-universe-main-dev
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
Environment: 
Nodes: 1
Port: 9200 (default Elasticsearch port)
Coordinator endpoint: http://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

Checking if Elasticsearch is already installed for environment: ...
Validating Elasticsearch installation...
./run.sh: line 132: /var/www/html/skeleton.dev.fastorder.com/fixing/scripts/lib/elasticsearch_validator.sh: No such file or directory
⚠️  Elasticsearch installation issues detected. Attempting automatic repair...
./run.sh: line 134: /var/www/html/skeleton.dev.fastorder.com/fixing/scripts/lib/elasticsearch_validator.sh: No such file or directory
Executing: steps/01-setup-directories.sh
+ 01-setup-directories.sh:4:main: echo '=== Step 1: Creating directory structure ==='
=== Step 1: Creating directory structure ===
+++ 01-setup-directories.sh:4:main: dirname steps/01-setup-directories.sh
++ 01-setup-directories.sh:4:main: cd steps
++ 01-setup-directories.sh:4:main: pwd
+ 01-setup-directories.sh:4:main: SCRIPT_DIR=/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps
+ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/lib/provisioning-init.sh
++ 01-setup-directories.sh:4:main: RED='\033[0;31m'
++ 01-setup-directories.sh:4:main: GREEN='\033[0;32m'
++ 01-setup-directories.sh:4:main: YELLOW='\033[1;33m'
++ 01-setup-directories.sh:4:main: BLUE='\033[0;34m'
++ 01-setup-directories.sh:4:main: NC='\033[0m'
++ 01-setup-directories.sh:4:main: export TERM=dumb
++ 01-setup-directories.sh:4:main: TERM=dumb
++ 01-setup-directories.sh:4:main: export DEBIAN_FRONTEND=noninteractive
++ 01-setup-directories.sh:4:main: DEBIAN_FRONTEND=noninteractive
++ 01-setup-directories.sh:4:main: export NEEDRESTART_MODE=a
++ 01-setup-directories.sh:4:main: NEEDRESTART_MODE=a
++ 01-setup-directories.sh:4:main: export NEEDRESTART_SUSPEND=1
++ 01-setup-directories.sh:4:main: NEEDRESTART_SUSPEND=1
++ 01-setup-directories.sh:4:main: export DEBIAN_PRIORITY=critical
++ 01-setup-directories.sh:4:main: DEBIAN_PRIORITY=critical
++ 01-setup-directories.sh:4:main: export UCF_FORCE_CONFFOLD=1
++ 01-setup-directories.sh:4:main: UCF_FORCE_CONFFOLD=1
++ 01-setup-directories.sh:4:main: export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
++ 01-setup-directories.sh:4:main: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
++ 01-setup-directories.sh:4:main: [[ -n '' ]]
++ 01-setup-directories.sh:4:main: [[ -n /opt/fastorder/bash/scripts/env_app_setup/state ]]
++ 01-setup-directories.sh:4:main: [[ -d /opt/fastorder/bash/scripts/env_app_setup/state ]]
++ 01-setup-directories.sh:4:main: STATE_DIR=/opt/fastorder/bash/scripts/env_app_setup/state
++ 01-setup-directories.sh:4:main: export STATE_DIR
++ 01-setup-directories.sh:4:main: [[ -f /opt/fastorder/bash/scripts/env_app_setup/setup/setup.json ]]
++ 01-setup-directories.sh:4:main: SETUP_JSON=/opt/fastorder/bash/scripts/env_app_setup/setup/setup.json
++ 01-setup-directories.sh:4:main: FO_WRAPPER=/usr/local/bin/fastorder-provisioning-wrapper.sh
++ 01-setup-directories.sh:4:main: HTTP_PORT_BASE=9200
++ 01-setup-directories.sh:4:main: TRANSPORT_PORT_BASE=9300
++ 01-setup-directories.sh:4:main: PG_PORT_BASE=5432
++ 01-setup-directories.sh:4:main: APP_IP_SUBNETS=(['observability']='10.100.5' ['obs']='10.100.5' ['prometheus']='10.100.5' ['grafana']='10.100.5' ['loki']='10.100.5' ['tempo']='10.100.5' ['postgresql']='10.100.10' ['postgres']='10.100.10' ['pg']='10.100.10' ['elasticsearch']='10.100.20' ['es']='10.100.20' ['kafka']='10.100.30' ['redis']='10.100.40' ['mongodb']='10.100.50' ['mongo']='10.100.50' ['iam']='10.100.60' ['keycloak']='10.100.60' ['general']='10.100.1')
++ 01-setup-directories.sh:4:main: declare -A APP_IP_SUBNETS
++ 01-setup-directories.sh:4:main: APP_IP_RESERVED_START=(['observability']='2' ['postgresql']='2' ['elasticsearch']='2' ['kafka']='2' ['redis']='2' ['mongodb']='2' ['iam']='2' ['general']='50')
++ 01-setup-directories.sh:4:main: declare -A APP_IP_RESERVED_START
++ 01-setup-directories.sh:4:main: APP_IP_RESERVED_END=(['observability']='49' ['postgresql']='254' ['elasticsearch']='254' ['kafka']='254' ['redis']='254' ['mongodb']='254' ['iam']='254' ['general']='250')
++ 01-setup-directories.sh:4:main: declare -A APP_IP_RESERVED_END
+++ 01-setup-directories.sh:4:main: dirname /opt/fastorder/bash/scripts/env_app_setup/lib/provisioning-init.sh
++ 01-setup-directories.sh:4:main: _CONFIG_MGMT_LIB=/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
++ 01-setup-directories.sh:4:main: [[ -f /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh ]]
++ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
+++ 01-setup-directories.sh:4:main: set -Eeuo pipefail
+++ 01-setup-directories.sh:4:main: : /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
+++ 01-setup-directories.sh:4:main: STATE_DIR=/opt/fastorder/bash/scripts/env_app_setup/state
++ 01-setup-directories.sh:4:main: [[ /opt/fastorder/bash/scripts/env_app_setup/lib/provisioning-init.sh == \s\t\e\p\s\/\0\1\-\s\e\t\u\p\-\d\i\r\e\c\t\o\r\i\e\s\.\s\h ]]
++ 01-setup-directories.sh:4:main: set +e
++ 01-setup-directories.sh:4:main: set +u
++ 01-setup-directories.sh:4:main: set +o pipefail
++ 01-setup-directories.sh:4:main: set +E
+ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh
++ 01-setup-directories.sh:4:main: [[ /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh == \s\t\e\p\s\/\0\1\-\s\e\t\u\p\-\d\i\r\e\c\t\o\r\i\e\s\.\s\h ]]
+ 01-setup-directories.sh:4:main: init_environment
+ 01-setup-directories.sh:4:main: require_bin jq
+ 01-setup-directories.sh:4:main: for b in "$@"
+ 01-setup-directories.sh:4:main: command -v jq
+ 01-setup-directories.sh:4:main: local app_type=general
+ 01-setup-directories.sh:4:main: ENV_ID=user-universe-main-dev
+ 01-setup-directories.sh:4:main: [[ -z user-universe-main-dev ]]
+ 01-setup-directories.sh:4:main: [[ -z user-universe-main-dev ]]
+ 01-setup-directories.sh:4:main: ENV_ID=user-universe-main-dev
+ 01-setup-directories.sh:4:main: [[ -z user-universe-main-dev ]]
+ 01-setup-directories.sh:4:main: [[ -z user-universe-main-dev ]]
+ 01-setup-directories.sh:4:main: [[ -z user-universe-main-dev ]]
++ 01-setup-directories.sh:4:main: env_dir_for user-universe-main-dev
++ 01-setup-directories.sh:4:main: echo /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev
+ 01-setup-directories.sh:4:main: ENV_DIR=/opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev
++ 01-setup-directories.sh:4:main: topo_path_for user-universe-main-dev
++ 01-setup-directories.sh:4:main: echo /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: TOPOLOGY_JSON=/opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: [[ ! -f /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json ]]
+ 01-setup-directories.sh:4:main: validate_topology_json /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: local topo=/opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: [[ -r /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json ]]
+ 01-setup-directories.sh:4:main: jq -e '
    .schema_version == 1
    and (.general.id        | type=="string")
    and (.general.shared_ip | type=="string")
    and (.general.service   | type=="string")
    and (.general.zone    | type=="string")
    and (.general.env       | type=="string")
  ' /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
++ 01-setup-directories.sh:4:main: jq -r .general.service /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: SERVICE=user
++ 01-setup-directories.sh:4:main: jq -r .general.zone /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: zone=universe
++ 01-setup-directories.sh:4:main: jq -r .general.branch /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: BRANCH=main
++ 01-setup-directories.sh:4:main: jq -r .general.env /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: ENV=dev
++ 01-setup-directories.sh:4:main: jq -r '.general.es_nodes_num // 3' /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: ES_NODES_NUM=1
++ 01-setup-directories.sh:4:main: jq -r '.general.pg_workers_num // 3' /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: PG_WORKERS_NUM=1
++ 01-setup-directories.sh:4:main: jq -r '.general.pg_WORKERS_STANDBY_NUM // 3' /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: PG_WORKERS_STANDBY_NUM=3
++ 01-setup-directories.sh:4:main: jq -r '.general.pg_citus_enabled // "yes"' /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: PG_CITUS_ENABLED=yes
+ 01-setup-directories.sh:4:main: [[ general != \g\e\n\e\r\a\l ]]
++ 01-setup-directories.sh:4:main: jq -r .general.shared_ip /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: VM_IP=142.93.238.16
+ 01-setup-directories.sh:4:main: [[ general != \g\e\n\e\r\a\l ]]
++ 01-setup-directories.sh:4:main: jq -r '.general.shared_iface // empty' /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: IFACE=eth0:16
+ 01-setup-directories.sh:4:main: local FINAL_VM_IP=142.93.238.16
+ 01-setup-directories.sh:4:main: set -a
+ 01-setup-directories.sh:4:main: [[ -r /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/generated/general.env ]]
+ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/generated/general.env
++ 01-setup-directories.sh:4:main: ENV_ID=user-universe-main-dev
++ 01-setup-directories.sh:4:main: SERVICE=user
++ 01-setup-directories.sh:4:main: zone=universe
++ 01-setup-directories.sh:4:main: BRANCH=main
++ 01-setup-directories.sh:4:main: ENV=dev
++ 01-setup-directories.sh:4:main: VM_IP=142.93.238.16
++ 01-setup-directories.sh:4:main: IFACE=eth0:16
++ 01-setup-directories.sh:4:main: ROOT_DIR=/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
++ 01-setup-directories.sh:4:main: ENV_DIR=/opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev
++ 01-setup-directories.sh:4:main: TOPOLOGY_JSON=/opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
++ 01-setup-directories.sh:4:main: LOG_LEVEL=info
++ 01-setup-directories.sh:4:main: DEBUG_MODE=false
+ 01-setup-directories.sh:4:main: set +a
+ 01-setup-directories.sh:4:main: VM_IP=142.93.238.16
+ 01-setup-directories.sh:4:main: export ENV_ID SERVICE zone BRANCH ENV VM_IP IFACE ENV_DIR TOPOLOGY_JSON
+ 01-setup-directories.sh:4:main: export ES_NODES_NUM PG_WORKERS_NUM PG_WORKERS_STANDBY_NUM PG_CITUS_ENABLED
+ 01-setup-directories.sh:4:main: [[ general != \g\e\n\e\r\a\l ]]
+ 01-setup-directories.sh:4:main: info 'Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)'
+ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)'
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
+ 01-setup-directories.sh:4:main: return 0
+ 01-setup-directories.sh:4:main: SERVICE=user
+ 01-setup-directories.sh:4:main: ZONE=universe
+ 01-setup-directories.sh:4:main: BRANCH=main
+ 01-setup-directories.sh:4:main: ENV=dev
++ 01-setup-directories.sh:4:main: env_id
++ 01-setup-directories.sh:4:main: '[' user = auth ']'
++ 01-setup-directories.sh:4:main: '[' user = item ']'
++ 01-setup-directories.sh:4:main: echo user-universe-main-dev
+ 01-setup-directories.sh:4:main: ENV_ID=user-universe-main-dev
+ 01-setup-directories.sh:4:main: env=user-universe-main-dev
+ 01-setup-directories.sh:4:main: nodes=1
+ 01-setup-directories.sh:4:main: [[ 1 =~ ^[1-9][0-9]*$ ]]
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /etc/elasticsearch
[2026-01-12 20:38:05 UTC] USER=www-data EUID=0 PID=693608 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /var/lib/elasticsearch
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /var/log/elasticsearch
[2026-01-12 20:38:05 UTC] USER=www-data EUID=0 PID=693626 ACTION=fsop ARGS=mkdir -p /var/log/elasticsearch
+ 01-setup-directories.sh:4:main: APP_NAME=search
+ 01-setup-directories.sh:4:main: TOPOLOGY_FILE=/opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: command -v jq
+ 01-setup-directories.sh:4:main: [[ -f /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json ]]
++ 01-setup-directories.sh:4:main: jq -r --arg app search '.applications[$app].vm_ip // empty' /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: COORD_IP=10.100.1.218
+ 01-setup-directories.sh:4:main: [[ -z 10.100.1.218 ]]
+ 01-setup-directories.sh:4:main: [[ 10.100.1.218 == \n\u\l\l ]]
++ 01-setup-directories.sh:4:main: get_application_domain search
++ 01-setup-directories.sh:4:main: local app_type=search
++ 01-setup-directories.sh:4:main: [[ search == \g\e\n\e\r\a\l ]]
++ 01-setup-directories.sh:4:main: jq -r --arg app search '.applications[$app].domain // empty' /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: COORD_DOMAIN=search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com
+ 01-setup-directories.sh:4:main: info 'Coordinator exists: search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.218)'
+ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Coordinator exists: search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.218)'
[INFO] Coordinator exists: search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.218)
+ 01-setup-directories.sh:4:main: (( i=1 ))
+ 01-setup-directories.sh:4:main: (( i<=nodes ))
++ 01-setup-directories.sh:4:main: printf %02d 1
+ 01-setup-directories.sh:4:main: node_num=01
+ 01-setup-directories.sh:4:main: IDENTIFIER=node-01
+ 01-setup-directories.sh:4:main: APP_NAME=search-node-01
+ 01-setup-directories.sh:4:main: read -r NODE_IP NODE_DOMAIN
++ 01-setup-directories.sh:4:main: setup_directories_per_node node-01 search-node-01
++ 01-setup-directories.sh:4:main: local IDENTIFIER=node-01
++ 01-setup-directories.sh:4:main: local APP_NAME=search-node-01
++ 01-setup-directories.sh:4:main: local env
+++ 01-setup-directories.sh:4:main: env_id
+++ 01-setup-directories.sh:4:main: '[' user = auth ']'
+++ 01-setup-directories.sh:4:main: '[' user = item ']'
+++ 01-setup-directories.sh:4:main: echo user-universe-main-dev
++ 01-setup-directories.sh:4:main: env=user-universe-main-dev
++ 01-setup-directories.sh:4:main: local TOPOLOGY_FILE=/opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
++ 01-setup-directories.sh:4:main: info 'Setting up Elasticsearch node: node-01'
++ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Setting up Elasticsearch node: node-01'
++ 01-setup-directories.sh:4:main: local NODE_IP NODE_DOMAIN
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /etc/elasticsearch/user-universe-main-dev/node-01 /etc/elasticsearch/user-universe-main-dev-node-01
+++ 01-setup-directories.sh:4:main: jq -r --arg app search-node-01 '.applications[$app].vm_ip // empty' /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
[2026-01-12 20:38:05 UTC] USER=www-data EUID=0 PID=693644 ACTION=fsop ARGS=ln -sfn /etc/elasticsearch/user-universe-main-dev/node-01 /etc/elasticsearch/user-universe-main-dev-node-01
++ 01-setup-directories.sh:4:main: NODE_IP=10.100.1.219
++ 01-setup-directories.sh:4:main: [[ -z 10.100.1.219 ]]
++ 01-setup-directories.sh:4:main: [[ 10.100.1.219 == \n\u\l\l ]]
+++ 01-setup-directories.sh:4:main: get_application_domain search-node-01
+++ 01-setup-directories.sh:4:main: local app_type=search-node-01
+++ 01-setup-directories.sh:4:main: [[ search-node-01 == \g\e\n\e\r\a\l ]]
+++ 01-setup-directories.sh:4:main: jq -r --arg app search-node-01 '.applications[$app].domain // empty' /opt/fastorder/bash/scripts/env_app_setup/state/user-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: [[ 1 -eq 1 ]]
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /etc/elasticsearch/user-universe-main-dev/node-01 /etc/elasticsearch/current
++ 01-setup-directories.sh:4:main: NODE_DOMAIN=search-user-universe-main-dev-elasticsearch-node-01.fastorder.com
++ 01-setup-directories.sh:4:main: info 'Using existing node-01: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.219)'
++ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Using existing node-01: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.219)'
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh: line 13: printf: write error: Broken pipe
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /etc/elasticsearch/user-universe-main-dev/node-01/certs
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /var/lib/elasticsearch/user-universe-main-dev/node-01 /var/lib/elasticsearch/current
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /var/lib/elasticsearch/user-universe-main-dev/node-01/tmp
[2026-01-12 20:38:05 UTC] USER=www-data EUID=0 PID=693674 ACTION=fsop ARGS=ln -sfn /var/lib/elasticsearch/user-universe-main-dev/node-01 /var/lib/elasticsearch/current
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /var/log/elasticsearch/user-universe-main-dev/node-01 /var/log/elasticsearch/current
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /var/log/elasticsearch/user-universe-main-dev/node-01
[2026-01-12 20:38:05 UTC] USER=www-data EUID=0 PID=693693 ACTION=fsop ARGS=ln -sfn /var/log/elasticsearch/user-universe-main-dev/node-01 /var/log/elasticsearch/current
+ 01-setup-directories.sh:4:main: (( i++ ))
+ 01-setup-directories.sh:4:main: (( i<=nodes ))
+ 01-setup-directories.sh:4:main: success 'Directory structure created for '\''user-universe-main-dev'\'' with 1 node(s).'
+ 01-setup-directories.sh:4:main: printf '[ OK ] %s\n' 'Directory structure created for '\''user-universe-main-dev'\'' with 1 node(s).'
[ OK ] Directory structure created for 'user-universe-main-dev' with 1 node(s).
Executing: steps/02-install-dependencies.sh
=== Step 2: Installing/Validating Elasticsearch (latest) ===
++ 01-setup-directories.sh:4:main: id -u elasticsearch
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-universe-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/user-universe-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chown -R elasticsearch:elasticsearch /var/log/elasticsearch/user-universe-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chmod 0750 /etc/elasticsearch/user-universe-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chmod 0750 /var/lib/elasticsearch/user-universe-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chmod 0750 /var/log/elasticsearch/user-universe-main-dev/node-01
++ 01-setup-directories.sh:4:main: info 'Created dirs for user-universe-main-dev/node-01 @ 10.100.1.219'
++ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Created dirs for user-universe-main-dev/node-01 @ 10.100.1.219'
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh: line 13: printf: write error: Broken pipe
++ 01-setup-directories.sh:4:main: printf '%s\n' 10.100.1.219
/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh: line 58: printf: write error: Broken pipe
++ 01-setup-directories.sh:4:main: printf '%s\n' search-user-universe-main-dev-elasticsearch-node-01.fastorder.com
/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh: line 59: printf: write error: Broken pipe
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
[INFO] Cleaning dpkg/apt locks...
[2026-01-12 20:38:06 UTC] USER=www-data EUID=0 PID=693781 ACTION=cleanup-dpkg-locks ARGS=
steps/02-install-dependencies.sh: line 16: 693779 Killed                  command sudo -n "$WRAP" cleanup-dpkg-locks
[2026-01-12 20:38:06 UTC] USER=www-data EUID=0 PID=693799 ACTION=fsop ARGS=chmod 0755 /etc/apt/keyrings
[INFO] apt-get update…
[2026-01-12 20:38:06 UTC] USER=www-data EUID=0 PID=693809 ACTION=pkg ARGS=update
Hit:1 http://apt.postgresql.org/pub/repos/apt jammy-pgdg InRelease
Hit:2 https://packages.microsoft.com/repos/azure-cli jammy InRelease
Hit:3 https://packages.confluent.io/deb/7.6 stable InRelease
Hit:4 https://apt.grafana.com stable InRelease
Hit:5 https://deb.nodesource.com/node_22.x nodistro InRelease
Hit:6 https://artifacts.elastic.co/packages/8.x/apt stable InRelease
Hit:7 https://ppa.launchpadcontent.net/ondrej/php/ubuntu jammy InRelease
Hit:8 https://mirrors.edge.kernel.org/ubuntu jammy InRelease
Get:9 https://mirrors.edge.kernel.org/ubuntu jammy-backports InRelease [127 kB]
Hit:10 https://packages.clickhouse.com/deb stable InRelease
Get:11 https://mirrors.edge.kernel.org/ubuntu jammy-security InRelease [129 kB]
Get:12 https://mirrors.edge.kernel.org/ubuntu jammy-updates InRelease [128 kB]
Hit:13 https://repos.citusdata.com/community/ubuntu jammy InRelease
Get:14 https://mirrors.edge.kernel.org/ubuntu jammy-security/main amd64 Packages [2903 kB]
Get:15 https://mirrors.edge.kernel.org/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB]
Get:16 https://mirrors.edge.kernel.org/ubuntu jammy-security/universe amd64 Packages [1008 kB]
Get:17 https://mirrors.edge.kernel.org/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.4 kB]
Get:18 https://mirrors.edge.kernel.org/ubuntu jammy-updates/main amd64 Packages [3164 kB]
Get:19 https://mirrors.edge.kernel.org/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.1 kB]
Get:20 https://mirrors.edge.kernel.org/ubuntu jammy-updates/universe amd64 Packages [1245 kB]
Get:21 https://mirrors.edge.kernel.org/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.1 kB]
Fetched 8791 kB in 2s (3620 kB/s)
Reading package lists...
[INFO] Installed version : 8.19.9
[INFO] Candidate version : 8.19.9
✅ Elasticsearch already at latest (or only) available version.
✅ Elasticsearch installation validated.
🎉 Dependencies installed and up-to-date.
Executing: steps/03-create-env-configs.sh
=== Step 3: Creating environment configurations (master + nodes, TLS, units) ===
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
Configuring env: user-universe-main-dev (nodes: 1, http: 9200, transport: 9300)
Using heap size: 1024m per node
[2026-01-12 20:38:25 UTC] USER=www-data EUID=0 PID=695181 ACTION=fsop ARGS=chown root:root /etc/default/elasticsearch
[2026-01-12 20:38:25 UTC] USER=www-data EUID=0 PID=695190 ACTION=fsop ARGS=chmod 0644 /etc/default/elasticsearch
[2026-01-12 20:38:26 UTC] USER=www-data EUID=0 PID=695208 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/jvm.options
[2026-01-12 20:38:26 UTC] USER=www-data EUID=0 PID=695217 ACTION=fsop ARGS=chmod 0644 /etc/elasticsearch/jvm.options
[2026-01-12 20:38:26 UTC] USER=www-data EUID=0 PID=695235 ACTION=fsop ARGS=mkdir -p /etc/systemd/system.conf.d /etc/systemd/user.conf.d
[2026-01-12 20:38:26 UTC] USER=www-data EUID=0 PID=695262 ACTION=passthru ARGS=systemctl daemon-reload
Current max_map_count: 262144
Current swappiness:   1
[2026-01-12 20:38:26 UTC] USER=www-data EUID=0 PID=695350 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/log4j2.properties
[2026-01-12 20:38:26 UTC] USER=www-data EUID=0 PID=695364 ACTION=fsop ARGS=chmod 0644 /etc/elasticsearch/log4j2.properties
[2026-01-12 20:38:26 UTC] USER=www-data EUID=0 PID=695374 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-universe-main-dev/template
[2026-01-12 20:38:26 UTC] USER=www-data EUID=0 PID=695383 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/user-universe-main-dev /etc/elasticsearch/user-universe-main-dev/template
[2026-01-12 20:38:26 UTC] USER=www-data EUID=0 PID=695393 ACTION=fsop ARGS=chmod 0755 /etc/elasticsearch/user-universe-main-dev
[2026-01-12 20:38:26 UTC] USER=www-data EUID=0 PID=695402 ACTION=fsop ARGS=cp /etc/elasticsearch/jvm.options /etc/elasticsearch/user-universe-main-dev/template/jvm.options
[INFO] 🌐 Registering general environment domain: user-universe-main-dev.fastorder.com
[INFO]   Allocated VM IP: 10.100.1.50 for general environment
[INFO]   Configuring VM IP 10.100.1.50 on network interface...
[WARNING]   VM IP may already be configured or need manual setup
[WARNING]   Warning: VM IP 10.100.1.50 not found on network interfaces
[2026-01-12 20:38:27 UTC] USER=www-data EUID=0 PID=695473 ACTION=fsop ARGS=sed -i /\suser-universe-main-dev.fastorder.com\(\s\|\$\)/d /etc/hosts
[INFO]   Removed old entry for user-universe-main-dev.fastorder.com
[ OK ] ✅ Registered general domain user-universe-main-dev.fastorder.com -> 10.100.1.50
[ OK ] ✅ DNS resolution verified for user-universe-main-dev.fastorder.com
[INFO] → Configuring user-universe-main-dev-node-01 (10.100.1.219) roles=[ master, data, data_hot, data_content, ingest ]
[2026-01-12 20:38:27 UTC] USER=www-data EUID=0 PID=695509 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-universe-main-dev/node-01/certs /var/lib/elasticsearch/user-universe-main-dev/node-01/tmp /var/log/elasticsearch/user-universe-main-dev/node-01
[2026-01-12 20:38:27 UTC] USER=www-data EUID=0 PID=695518 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-universe-main-dev/node-01
[2026-01-12 20:38:27 UTC] USER=www-data EUID=0 PID=695527 ACTION=fsop ARGS=chmod 0750 /etc/elasticsearch/user-universe-main-dev/node-01 /var/lib/elasticsearch/user-universe-main-dev/node-01 /var/log/elasticsearch/user-universe-main-dev/node-01
[2026-01-12 20:38:27 UTC] USER=www-data EUID=0 PID=695536 ACTION=fsop ARGS=cp /etc/elasticsearch/user-universe-main-dev/template/jvm.options /etc/elasticsearch/user-universe-main-dev/node-01/jvm.options
[2026-01-12 20:38:27 UTC] USER=www-data EUID=0 PID=695545 ACTION=fsop ARGS=sed -i s/^-Xms.*/-Xms1024m/ /etc/elasticsearch/user-universe-main-dev/node-01/jvm.options
[2026-01-12 20:38:27 UTC] USER=www-data EUID=0 PID=695554 ACTION=fsop ARGS=sed -i s/^-Xmx.*/-Xmx1024m/ /etc/elasticsearch/user-universe-main-dev/node-01/jvm.options
[2026-01-12 20:38:27 UTC] USER=www-data EUID=0 PID=695611 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-universe-main-dev/node-01
[2026-01-12 20:38:27 UTC] USER=www-data EUID=0 PID=695620 ACTION=fsop ARGS=chmod 0644 /etc/elasticsearch/user-universe-main-dev/node-01/elasticsearch.yml
[2026-01-12 20:38:27 UTC] USER=www-data EUID=0 PID=695638 ACTION=fsop ARGS=chmod 0644 /etc/default/elasticsearch-user-universe-main-dev-node-01
[2026-01-12 20:38:27 UTC] USER=www-data EUID=0 PID=695665 ACTION=passthru ARGS=ip addr add 10.100.1.219/32 dev eth0 label eth0:219
[2026-01-12 20:38:27 UTC] USER=www-data EUID=0 PID=695674 ACTION=fsop ARGS=sed -i /[[:space:]]search-user-universe-main-dev-elasticsearch-node-01.fastorder.com\([[:space:]]\|$\)/d /etc/hosts
[INFO] ✔ Created configuration for user-universe-main-dev/node-01 (roles=single-node)
[2026-01-12 20:38:27 UTC] USER=www-data EUID=0 PID=695695 ACTION=fsop ARGS=sed -i /[[:space:]]search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com\([[:space:]]\|$\)/d /etc/hosts
[INFO] ✔ Registered master domain search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com -> 10.100.1.219 (points to node-01)
[INFO] Cleaning up legacy non-templated elasticsearch-*.service units (if any)...
[INFO] No legacy units found.
[INFO] Base template exists: elasticsearch@.service
[INFO] Unit exists: elasticsearch@user-universe-main-dev-node-01.service
[2026-01-12 20:38:28 UTC] USER=www-data EUID=0 PID=695732 ACTION=passthru ARGS=systemctl daemon-reload

[ OK ] Environment configurations (master + nodes with TLS) created successfully!
[INFO] Environment: user-universe-main-dev
[INFO] Nodes: 1
[INFO] HTTP Port: 9200
[INFO] Transport Port: 9300
[INFO] Heap Size: 1024m per node
[INFO] Master: search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.218)
[INFO]   node-01: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.219)
[INFO] Systemd units prepared (not started). Start sequence runs in Step 7.
Executing: steps/04-start-clusters.sh
=== Step 7: Starting Elasticsearch clusters (with waits) ===
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
Starting Elasticsearch cluster for environment: user-universe-main-dev (1 nodes)
[INFO] === Ensuring VM IP services are started ===
[WARNING] VM IP service vm-ip-10-100-1-219.service not found - IP might not persist
[INFO] Manually configuring IP: 10.100.1.219
[2026-01-12 20:38:30 UTC] USER=www-data EUID=0 PID=695902 ACTION=configure-network-interface ARGS=lo:search01 10.100.1.219
✓ lo:search01 <- 10.100.1.219
[INFO] Cleaning up any existing Elasticsearch processes and lock files...
[2026-01-12 20:38:30 UTC] USER=www-data EUID=0 PID=695921 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@user-universe-main-dev-node-01.service
[INFO] Stopping existing service: elasticsearch@user-universe-main-dev-node-01.service
[2026-01-12 20:38:30 UTC] USER=www-data EUID=0 PID=695938 ACTION=passthru ARGS=systemctl stop elasticsearch@user-universe-main-dev-node-01.service
[INFO] Stopping Elasticsearch services for environment: user-universe-main-dev ...
[INFO] No active Elasticsearch services found for environment: user-universe-main-dev
[INFO] Removing lock files from: /var/lib/elasticsearch/user-universe-main-dev/node-01
[2026-01-12 20:38:33 UTC] USER=www-data EUID=0 PID=695995 ACTION=fsop ARGS=find /var/lib/elasticsearch/user-universe-main-dev/node-01 -name *.lock -delete
[2026-01-12 20:38:33 UTC] USER=www-data EUID=0 PID=696004 ACTION=fsop ARGS=find /var/lib/elasticsearch/user-universe-main-dev/node-01 -name node.lock -delete
[2026-01-12 20:38:33 UTC] USER=www-data EUID=0 PID=696017 ACTION=fsop ARGS=find /var/lib/elasticsearch/user-universe-main-dev/node-01 -name _state -type d -exec rm -rf {} +
[2026-01-12 20:38:33 UTC] USER=www-data EUID=0 PID=696027 ACTION=fsop ARGS=find /tmp -name *elasticsearch*user-universe-main-dev-node-01* -delete
[ OK ] Cleanup completed for environment: user-universe-main-dev
[INFO] Checking for port conflicts before starting Elasticsearch...
[INFO] Checking for port conflicts on 10.100.1.218:9200 and 10.100.1.218:9300...
[ OK ] ✓ Ports 9200 and 9300 are available on 10.100.1.218
[INFO] Ensuring correct ownership of Elasticsearch directories...
[2026-01-12 20:38:35 UTC] USER=www-data EUID=0 PID=696102 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch
[2026-01-12 20:38:36 UTC] USER=www-data EUID=0 PID=696121 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /var/lib/elasticsearch
[2026-01-12 20:38:37 UTC] USER=www-data EUID=0 PID=696145 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /var/log/elasticsearch
[ OK ] Directory ownership fixed
[INFO] === Starting Elasticsearch Nodes ===
[INFO] Starting 1 node(s) for cluster
▶ Starting elasticsearch@user-universe-main-dev-node-01.service (search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200)
[2026-01-12 20:38:37 UTC] USER=www-data EUID=0 PID=696158 ACTION=passthru ARGS=systemctl is-enabled --quiet elasticsearch@user-universe-main-dev-node-01.service
[2026-01-12 20:38:37 UTC] USER=www-data EUID=0 PID=696167 ACTION=passthru ARGS=systemctl start elasticsearch@user-universe-main-dev-node-01.service
⏳ Waiting for TCP search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200 to be accessible (timeout 360s)...
✅ Port 9200 is accessible on search-user-universe-main-dev-elasticsearch-node-01.fastorder.com.
⏳ Waiting for ES HTTP readiness on http://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200 (timeout 300s)...
[ OK ] ES HTTP ready on search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
[ OK ] elasticsearch@user-universe-main-dev-node-01.service is up and answering HTTP on search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Node 1 started successfully
[INFO] Cluster with 1 node(s) started successfully
⏳ Waiting for the cluster to elect master and settle...
⏳ Waiting for cluster health=green via search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200 (timeout 300s)...
[ OK ] Cluster is GREEN (nodes="number_of_nodes") on search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
[ OK ] Cluster user-universe-main-dev is healthy and green!

[INFO] === Final Status Check ===
[2026-01-12 20:39:18 UTC] USER=www-data EUID=0 PID=697456 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@user-universe-main-dev-node-01.service
[ OK ] elasticsearch@user-universe-main-dev-node-01.service is ACTIVE (search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200)
  └── HTTP responding on search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200 ✓
[ OK ] All 1 node(s) in environment 'user-universe-main-dev' are running successfully!
[INFO] Node endpoints:
  - http://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200


[ OK ] Elasticsearch cluster started successfully!
[INFO] Environment: user-universe-main-dev
[INFO] Nodes: 1
[INFO] Cluster endpoints:
  - http://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === Quick Cluster Information ===
Cluster Name: fastorder-user-universe-main-dev
Node Name: user-universe-main-dev-node-01
Version: 8.19.9
Architecture: 1 node(s), each on default port 9200

Cluster with 1 node(s) started successfully (each on port 9200)
Executing: steps/05-verify-setup.sh
=== Step 8: Verifying setup (with retries) ===
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
Verifying environment: user-universe-main-dev (1 nodes, Single-node)
Main HTTP endpoint: http://search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
Testing network connectivity to search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200...
✓ Domain connection available
Testing HTTP response...
[ OK ] ✓ user-universe-main-dev is responding on search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200

[INFO] === Cluster Health ===
{
  "cluster_name" : "fastorder-user-universe-main-dev",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
[ OK ] Cluster status: GREEN ("number_of_nodes" nodes)

[INFO] === Cluster Nodes ===
ip           heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.100.1.219           53          98  80    4.65    4.30     4.22 dhims     *      user-universe-main-dev-node-01

[INFO] === Single-Node Service Verification ===
Testing coordinator service (search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200)...
  ✓ Coordinator HTTP responding on search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
    Name: user-universe-main-dev-node-01, Version: 8.19.9

[INFO] === Cluster State Summary ===
Using jq for formatted output:
jq parsing failed

[ OK ] === Verification Summary ===
[INFO] Environment: user-universe-main-dev
[INFO] Nodes configured: 1
[INFO] Main endpoint: http://search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
[INFO] Service endpoint: http://search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200

[INFO] === Final Connectivity Test ===
  ✓ Coordinator: search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200

[ OK ] Single-node cluster is responding successfully!
[ OK ] Elasticsearch cluster 'user-universe-main-dev' verification completed successfully!
Executing: steps/06-confirm-working.sh
=== Step 9: Comprehensive Cluster Verification (gated) ===
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
========================================
🔍 Verifying Environment: user-universe-main-dev (1 nodes)
========================================
Domain: search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com
Environment: user-universe-main-dev
Nodes: 1

[INFO] Testing network connectivity...
Setup type: Single-node
Testing endpoint: search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
[ OK ] ✓ Using domain: search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com


📡 Coordinator Service (elasticsearch@user-universe-main-dev-node-01.service)
Endpoint: search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
--------------------------------
[2026-01-12 20:39:19 UTC] USER=www-data EUID=0 PID=697605 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@user-universe-main-dev-node-01.service
✅ Service: ACTIVE
⏳ Waiting for TCP search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200 to be accessible (timeout 5s)...
✅ Port 9200 is accessible on search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com.
✅ Port: LISTENING on search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
✅ HTTP: RESPONDING on search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
   Node name: user-universe-main-dev-node-01

========================================
🏥 Cluster Health Check
========================================
Cluster Name: fastorder-user-universe-main-dev
Nodes Count: "number_of_nodes"
Status: green
[ OK ] ✅ Cluster status: GREEN (healthy)

Full cluster health:
{
  "cluster_name" : "fastorder-user-universe-main-dev",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

========================================
📊 Final Verification Results
========================================
[ OK ] ✅ Comprehensive verification PASSED!
[ OK ] Environment 'user-universe-main-dev' with 1 nodes is fully operational

📋 QUICK DIAGNOSTIC COMMANDS:
----------------------------------------
# Test cluster endpoints:
curl http://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

# Check cluster health:
curl http://search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200/_cluster/health?pretty

# Check nodes info:
curl http://search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200/_cat/nodes?v

# Check all Elasticsearch ports:
sudo ss -tlnp | grep java

# Check systemd service status:
sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status elasticsearch@user-universe-main-dev-node-01.service

# View recent logs:
sudo journalctl -u elasticsearch@user-universe-main-dev-node-01.service -f

[INFO] Environment: user-universe-main-dev
[INFO] Nodes: 1
[INFO] Port: 9200 (default Elasticsearch port)
[INFO] Coordinator endpoint: http://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

=== Elasticsearch HTTP Setup completed successfully! ===
Environment:  (1 nodes)
Port: 9200 (default Elasticsearch port)

✅ Coordinator endpoint: http://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

Quick test commands:
  curl http://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
  curl http://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty


✓ Step 1 completed successfully!

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 2: Executing Make Https
Folder: 02-make-https
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

=== Elasticsearch HTTPS Setup ===
Configure HTTPS/SSL for Elasticsearch cluster
[INFO] Using web-provided environment: user-universe-main-dev
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
Environment: 
Nodes: 1
Node: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.219)
Port: 9200 (default port)

Executing: steps/01-generate-ssl-certificates.sh
==================================================================
STEP 1: Generate SSL certificates for Elasticsearch transport
==================================================================
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
Domain: user-universe-main-dev.fastorder.com
Environment: user-universe-main-dev
Nodes: 1
Per-node VM IPs and domains:
  Node 1: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.219)
Port: 9200 (default port for all nodes)

=== Generating SSL certificates for ES transport ===
[INFO] Generating certificates for environment: user-universe-main-dev (1 nodes)
[INFO] Configuring certificates for 1 node(s)
[INFO] Certificate storage: /etc/fastorder/elasticsearch/certs/user-universe-main-dev
[2026-01-12 20:39:20 UTC] USER=www-data EUID=0 PID=697696 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/temp-697673
[2026-01-12 20:39:20 UTC] USER=www-data EUID=0 PID=697705 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/temp-697673
[2026-01-12 20:39:20 UTC] USER=www-data EUID=0 PID=697714 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/temp-697673
[2026-01-12 20:39:20 UTC] USER=www-data EUID=0 PID=697733 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/temp-697673/instances.yml
[INFO] Creating certificate instances configuration...
  Adding node: user-universe-main-dev-node-01 (search-user-universe-main-dev-elasticsearch-node-01.fastorder.com, 10.100.1.219)

[INFO] Certificate instances configuration:
instances:
  - name: user-universe-main-dev-node-01
    dns: [ "user-universe-main-dev-node-01", "localhost", "search-user-universe-main-dev-elasticsearch-node-01.fastorder.com" ]
    ip:  [ "10.100.1.219", "127.0.0.1" ]

[INFO] Creating Certificate Authority for user-universe-main-dev...
[2026-01-12 20:39:20 UTC] USER=www-data EUID=0 PID=697758 ACTION=fsop ARGS=mkdir -p /etc/fastorder/elasticsearch/certs/user-universe-main-dev/node-01/certs
[2026-01-12 20:39:20 UTC] USER=www-data EUID=0 PID=697767 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/fastorder/elasticsearch/certs/user-universe-main-dev
[2026-01-12 20:39:20 UTC] USER=www-data EUID=0 PID=697776 ACTION=fsop ARGS=chmod -R 755 /etc/fastorder/elasticsearch/certs/user-universe-main-dev
[2026-01-12 20:39:20 UTC] USER=www-data EUID=0 PID=697785 ACTION=fsop ARGS=rm -f /etc/fastorder/elasticsearch/certs/user-universe-main-dev/node-01/certs/user-universe-main-dev-ca.zip
yes: standard output: Broken pipe
[ OK ] ✓ CA certificate created

[INFO] Creating node certificates for user-universe-main-dev...
yes: standard output: Broken pipe
[ OK ] ✓ Node certificates created

[INFO] Distributing certificates...
  Configuring certificates for node 1 (user-universe-main-dev-node-01)...
[2026-01-12 20:39:26 UTC] USER=www-data EUID=0 PID=697957 ACTION=fsop ARGS=cp /etc/fastorder/elasticsearch/certs/user-universe-main-dev/node-01/certs/ca/ca.crt /etc/fastorder/elasticsearch/certs/user-universe-main-dev/node-01/ca.crt
[2026-01-12 20:39:26 UTC] USER=www-data EUID=0 PID=697972 ACTION=fsop ARGS=cp /etc/fastorder/elasticsearch/certs/user-universe-main-dev/node-01/certs/nodes/user-universe-main-dev-node-01.crt /etc/fastorder/elasticsearch/certs/user-universe-main-dev/node-01/
[2026-01-12 20:39:26 UTC] USER=www-data EUID=0 PID=697982 ACTION=fsop ARGS=cp /etc/fastorder/elasticsearch/certs/user-universe-main-dev/node-01/certs/nodes/user-universe-main-dev-node-01.key /etc/fastorder/elasticsearch/certs/user-universe-main-dev/node-01/
[2026-01-12 20:39:26 UTC] USER=www-data EUID=0 PID=697997 ACTION=fsop ARGS=chmod 644 /etc/fastorder/elasticsearch/certs/user-universe-main-dev/node-01/user-universe-main-dev-node-01.crt
[2026-01-12 20:39:27 UTC] USER=www-data EUID=0 PID=698006 ACTION=fsop ARGS=chmod 600 /etc/fastorder/elasticsearch/certs/user-universe-main-dev/node-01/user-universe-main-dev-node-01.key
[ OK ]   ✓ Certificates copied for user-universe-main-dev-node-01
[2026-01-12 20:39:27 UTC] USER=www-data EUID=0 PID=698015 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/fastorder/elasticsearch/certs/user-universe-main-dev/node-01
[2026-01-12 20:39:27 UTC] USER=www-data EUID=0 PID=698026 ACTION=fsop ARGS=find /etc/fastorder/elasticsearch/certs/user-universe-main-dev/node-01/certs -type f -name *.key -exec chmod 600 {} ;
[2026-01-12 20:39:27 UTC] USER=www-data EUID=0 PID=698046 ACTION=fsop ARGS=rm -rf /etc/elasticsearch/temp-697673
[ OK ] ✓ Certificates ready for environment: user-universe-main-dev

[ OK ] ✓ SSL certificate generation completed successfully!
[INFO] Environment: user-universe-main-dev
[INFO] Nodes configured: 1
[INFO] Per-node VM IPs and domains (each with default port 9200):
  Node 1: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.219)
[INFO] Certificate directory: /etc/fastorder/elasticsearch/certs/user-universe-main-dev/node-01/certs

[INFO] === Certificate Summary ===
CA Certificate: /etc/fastorder/elasticsearch/certs/user-universe-main-dev/node-01/certs/ca/ca.crt
Node Certificates:
  - user-universe-main-dev-node-01: /etc/fastorder/elasticsearch/certs/user-universe-main-dev/node-01/

[INFO] === Verification Commands ===
# Verify CA certificate:
openssl x509 -in /etc/fastorder/elasticsearch/certs/user-universe-main-dev/node-01/certs/ca/ca.crt -text -noout

# Verify node certificates:
openssl x509 -in /etc/fastorder/elasticsearch/certs/user-universe-main-dev/node-01/user-universe-main-dev-node-01.crt -text -noout

[INFO] Next: Configure transport SSL in Elasticsearch configuration files
Executing: steps/02-enable-security-transport.sh
==================================================================
STEP 2: Enable security with transport SSL
==================================================================
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
Environment: user-universe-main-dev
Nodes: 1
Node: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.219)

[INFO] === Single-Node Setup ===
[INFO] Enabling security (xpack.security.enabled: true)
[2026-01-12 20:39:27 UTC] USER=www-data EUID=0 PID=698087 ACTION=fsop ARGS=sed -i /^xpack.security.enabled:/d /etc/elasticsearch/user-universe-main-dev/node-01/elasticsearch.yml
[INFO] Disabling transport SSL (not needed for single-node)
[2026-01-12 20:39:27 UTC] USER=www-data EUID=0 PID=698106 ACTION=fsop ARGS=sed -i /^xpack.security.transport.ssl.enabled:/d /etc/elasticsearch/user-universe-main-dev/node-01/elasticsearch.yml

[ OK ] ==================================================================
[ OK ] Security and Transport SSL Configuration Complete
[ OK ] ==================================================================
[INFO] Environment: user-universe-main-dev
[INFO] Nodes: 1
[INFO] Security enabled: true
[INFO] Transport SSL enabled: false (not required for single-node)

[INFO] === Next Step ===
Restart services to apply security configuration (step 04)
Executing: steps/03-http-ssl.sh
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
==================================================================
Direct HTTPS Configuration (native TLS, PEM)
==================================================================
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
Environment: user-universe-main-dev
Nodes: 1
Node: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.219)
Port: 9200 (default port)
Domain: user-universe-main-dev.fastorder.com

[INFO] === Single-Node Direct HTTPS Setup ===
🔐 Generated PKCS12 password: nQ0nIHAs... (32 chars)
[INFO] Using first node configuration: /etc/elasticsearch/user-universe-main-dev/node-01
[2026-01-12 20:39:28 UTC] USER=www-data EUID=0 PID=698185 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-universe-main-dev/node-01
[2026-01-12 20:39:28 UTC] USER=www-data EUID=0 PID=698194 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-universe-main-dev/node-01
[INFO] Checking prerequisites...
[ OK ] ✓ Prerequisites verified
[INFO] Setting up temporary directories...
[2026-01-12 20:39:28 UTC] USER=www-data EUID=0 PID=698205 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out /etc/elasticsearch/user-universe-main-dev/node-01/http-certs
[2026-01-12 20:39:28 UTC] USER=www-data EUID=0 PID=698214 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out /etc/elasticsearch/user-universe-main-dev/node-01/http-certs
[2026-01-12 20:39:28 UTC] USER=www-data EUID=0 PID=698223 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-universe-main-dev
[2026-01-12 20:39:28 UTC] USER=www-data EUID=0 PID=698232 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-universe-main-dev/node-01
[2026-01-12 20:39:28 UTC] USER=www-data EUID=0 PID=698241 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-universe-main-dev/node-01/http-certs
[2026-01-12 20:39:28 UTC] USER=www-data EUID=0 PID=698250 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out
[ OK ] ✓ Directories created

[INFO] Building certificate instances configuration...
[INFO] Certificate instances configuration:
instances:
  - name: "user-universe-main-dev-http"
    dns:  [ "localhost", "web-03", "search-user-universe-main-dev-elasticsearch-node-01.fastorder.com", "user-universe-main-dev-node-01.fastorder.com", "search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com", "search-user-universe-main-dev.fastorder.com", "user-universe-main-dev-node-01.local" ]
    ip:   [ "10.100.1.219", "127.0.0.1", "::1" ]
[ OK ] ✓ Instances configuration created

[INFO] Generating HTTP Certificate Authority...
[2026-01-12 20:39:28 UTC] USER=www-data EUID=0 PID=698269 ACTION=fsop ARGS=rm -f /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/http-ca.zip /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out/http-certs.zip
[2026-01-12 20:39:28 UTC] USER=www-data EUID=0 PID=698282 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-universe-main-dev/node-01/http-certs
[2026-01-12 20:39:28 UTC] USER=www-data EUID=0 PID=698291 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-universe-main-dev/node-01/http-certs
[2026-01-12 20:39:31 UTC] USER=www-data EUID=0 PID=698341 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/http-ca.zip
Archive:  /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/http-ca.zip
  inflating: /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/ca/ca.crt  
  inflating: /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/ca/ca.key  
[ OK ] ✓ HTTP CA generated successfully

[INFO] Generating per-node HTTP certificates...
[2026-01-12 20:39:31 UTC] USER=www-data EUID=0 PID=698353 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out
[2026-01-12 20:39:31 UTC] USER=www-data EUID=0 PID=698362 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out
[2026-01-12 20:39:33 UTC] USER=www-data EUID=0 PID=698446 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out/http-certs.zip
[2026-01-12 20:39:33 UTC] USER=www-data EUID=0 PID=698455 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out/http
Archive:  /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out/http-certs.zip
  inflating: /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out/http/user-universe-main-dev-http/user-universe-main-dev-http.crt  
  inflating: /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out/http/user-universe-main-dev-http/user-universe-main-dev-http.key  
[ OK ] ✓ HTTP certificates generated successfully

[INFO] Installing certificates and configuring services...
[INFO] Configuring main service for single-node HTTPS...
[2026-01-12 20:39:33 UTC] USER=www-data EUID=0 PID=698467 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-universe-main-dev/node-01/certs
[2026-01-12 20:39:33 UTC] USER=www-data EUID=0 PID=698476 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-universe-main-dev/node-01/certs
[2026-01-12 20:39:34 UTC] USER=www-data EUID=0 PID=698485 ACTION=fsop ARGS=cp /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out/http/user-universe-main-dev-http/user-universe-main-dev-http.crt /etc/elasticsearch/user-universe-main-dev/node-01/certs/http.crt
[2026-01-12 20:39:34 UTC] USER=www-data EUID=0 PID=698494 ACTION=fsop ARGS=cp /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out/http/user-universe-main-dev-http/user-universe-main-dev-http.key /etc/elasticsearch/user-universe-main-dev/node-01/certs/http.key
[2026-01-12 20:39:34 UTC] USER=www-data EUID=0 PID=698503 ACTION=fsop ARGS=cp /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/ca/ca.crt /etc/elasticsearch/user-universe-main-dev/node-01/certs/http_ca.crt
[2026-01-12 20:39:34 UTC] USER=www-data EUID=0 PID=698512 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/user-universe-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-01-12 20:39:34 UTC] USER=www-data EUID=0 PID=698530 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/user-universe-main-dev/node-01/certs/http.key
[2026-01-12 20:39:34 UTC] USER=www-data EUID=0 PID=698539 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/user-universe-main-dev/node-01/certs/http.crt /etc/elasticsearch/user-universe-main-dev/node-01/certs/http_ca.crt /etc/elasticsearch/user-universe-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-01-12 20:39:34 UTC] USER=www-data EUID=0 PID=698548 ACTION=fsop ARGS=sed -i -E -e /^\s*xpack\.security\.http\.ssl(\..*)?\s*:/d /etc/elasticsearch/user-universe-main-dev/node-01/elasticsearch.yml
[2026-01-12 20:39:34 UTC] USER=www-data EUID=0 PID=698557 ACTION=fsop ARGS=sed -i /^# --- BEGIN direct HTTPS (managed, PEM) ---$/,/^# --- END direct HTTPS (managed, PEM) ---$/d /etc/elasticsearch/user-universe-main-dev/node-01/elasticsearch.yml
[2026-01-12 20:39:34 UTC] USER=www-data EUID=0 PID=698566 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/user-universe-main-dev/node-01/certs/fastorder_ra_root.crt
YAML: /etc/elasticsearch/user-universe-main-dev/node-01/elasticsearch.yml
[ OK ]   ✓ Main service configured with HTTPS
[2026-01-12 20:39:34 UTC] USER=www-data EUID=0 PID=698585 ACTION=fsop ARGS=rm -rf /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients
[2026-01-12 20:39:34 UTC] USER=www-data EUID=0 PID=698594 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients
[2026-01-12 20:39:34 UTC] USER=www-data EUID=0 PID=698603 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients
Archive:  /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client.zip
   creating: /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/
  inflating: /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.crt  
  inflating: /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.key  
[INFO] Creating P12 keystore for es-client...
[2026-01-12 20:39:36 UTC] USER=www-data EUID=0 PID=698679 ACTION=fsop ARGS=mv /tmp/es-client-698141.p12 /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[ OK ] ✓ Created P12 keystore: /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-01-12 20:39:36 UTC] USER=www-data EUID=0 PID=698688 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients
[2026-01-12 20:39:36 UTC] USER=www-data EUID=0 PID=698697 ACTION=fsop ARGS=chmod 640 /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.key
[2026-01-12 20:39:36 UTC] USER=www-data EUID=0 PID=698706 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-01-12 20:39:36 UTC] USER=www-data EUID=0 PID=698715 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.crt /etc/elasticsearch/user-universe-main-dev/node-01/certs/http_ca.crt
[INFO] Saving keystore passwords to secrets vault...
🔑 Configuring AWS credentials for secrets vault...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[INFO] 🔐 Vaulting search passwords to remote backend...
✅ Passwords vaulted to remote backend
✓ Keystore passwords saved to secrets vault: search/user-universe-main-dev/keystore-passwords

[INFO] === Installing CA Certificate for Users ===
[INFO] HOME not set, skipping user CA installation

✓ Direct HTTPS configuration completed for environment: user-universe-main-dev
[INFO] All services now serve HTTPS using PEM certificates
[INFO] Network binding: 10.100.1.219
[INFO] HTTPS endpoint: https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === Certificate Summary ===
CA Certificate: /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/ca/ca.crt
Certificate Directory: /etc/elasticsearch/user-universe-main-dev/node-01/http-certs
Main service certificates: /etc/elasticsearch/user-universe-main-dev/node-01/certs/

[INFO] === Next Steps ===
1. Restart Elasticsearch services to apply HTTPS configuration
2. Test HTTPS connectivity: curl https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
3. Verify certificates: openssl s_client -connect search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === SSL Certificate Information ===
• CA certificate installed for user - curl will work without -k flag
• Each certificate includes node-specific domain, VM IP, and localhost in Subject Alternative Names
• Certificates are valid for 3 years from generation date

[WARNING] Important: You'll need to restart Elasticsearch services for HTTPS to take effect
Executing: steps/04-restart-systemd-services.sh
==================================================================
STEP 4 (STRICT): Restart systemd services and verify secure health
==================================================================
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: user-universe-main-dev
Nodes: 1
Per-node endpoints (all use default port 9200):
  Node 1: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.219)

[INFO] Building service list for environment: user-universe-main-dev (1 nodes)
  - elasticsearch@user-universe-main-dev-node-01.service (port 9200)

[INFO] Will restart 1 service(s) for environment: user-universe-main-dev
[2026-01-12 20:39:39 UTC] USER=www-data EUID=0 PID=698864 ACTION=passthru ARGS=systemctl daemon-reload

[INFO] === Ensuring VM IPs are configured correctly ===
[INFO] ✓ 10.100.1.219 already configured on eth0 for node-01

[INFO] === Ensuring transport SSL certificates for all nodes ===
[INFO] ✓ Transport certificates already exist for node-01

[INFO] === Restarting Services ===
↻ Restarting elasticsearch@user-universe-main-dev-node-01.service ...
[2026-01-12 20:39:40 UTC] USER=www-data EUID=0 PID=698914 ACTION=passthru ARGS=systemctl restart elasticsearch@user-universe-main-dev-node-01.service
[2026-01-12 20:39:44 UTC] USER=www-data EUID=0 PID=699068 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@user-universe-main-dev-node-01.service
[ OK ] elasticsearch@user-universe-main-dev-node-01.service is active
[INFO] Waiting 10s for Elasticsearch to start listening on ports...

[INFO] === Waiting for STRICT Secure Cluster Health ===
[INFO] Waiting for port 9200 on 10.100.1.219 (timeout 120s)...
[INFO] Waiting for cluster to form and be ready for write operations...
✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
[INFO] Cluster stable and ready for operations (20 consecutive healthy responses over 40s)

✓ Retrieved password from AWS Secrets Manager
[INFO] Testing cluster at: https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Using SSL CA certificate: /etc/elasticsearch/user-universe-main-dev/node-01/certs/http_ca.crt
[INFO] Using client cert/key for mTLS
[INFO] Using client cert/key for mTLS: /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.key
[INFO]   ⏳ waiting for secure cluster health (require 200) at https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200 (timeout 30s)...
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] Cluster did not become healthy (secure 200) within 30s
[WARNING] Initial authentication failed - password may not be set in Elasticsearch yet
[WARNING] Running password setup to set/reset Elasticsearch password...
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
╔════════════════════════════════════════════════════════════╗
║   Elasticsearch Password Management via AWS Secrets MGR   ║
╚════════════════════════════════════════════════════════════╝

Environment: user-universe-main-dev
User:        elastic
Identifier:  node-01
AWS Secret:  fastorder/search/user/universe/main/dev/elasticsearch/node-01

Using configuration path: /etc/elasticsearch/user-universe-main-dev/node-01 (IDENTIFIER: node-01)
Node domain: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com
HTTP port: 9200 (default Elasticsearch port)
[INFO] xpack.security.enabled already true → no restart.
[INFO] No restart needed.
[2026-01-12 20:41:26 UTC] USER=www-data EUID=0 PID=701626 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-universe-main-dev/node-01
[2026-01-12 20:41:26 UTC] USER=www-data EUID=0 PID=701662 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/user-universe-main-dev/node-01/users /etc/elasticsearch/user-universe-main-dev/node-01/users_roles
[2026-01-12 20:41:26 UTC] USER=www-data EUID=0 PID=701681 ACTION=fsop ARGS=chmod 660 /etc/elasticsearch/user-universe-main-dev/node-01/users /etc/elasticsearch/user-universe-main-dev/node-01/users_roles
✓ users/users_roles present and writable
[2026-01-12 20:41:26 UTC] USER=www-data EUID=0 PID=701690 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/user-universe-main-dev/node-01/elasticsearch.keystore
[2026-01-12 20:41:26 UTC] USER=www-data EUID=0 PID=701699 ACTION=fsop ARGS=chmod 660 /etc/elasticsearch/user-universe-main-dev/node-01/elasticsearch.keystore
✓ Keystore exists: /etc/elasticsearch/user-universe-main-dev/node-01/elasticsearch.keystore
HTTPS is enabled in configuration
✓ Found HTTP CA certificate: /etc/elasticsearch/user-universe-main-dev/node-01/certs/http_ca.crt
✓ Using client certificates for mTLS
Waiting for Elasticsearch to be reachable at https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200...
✓ Elasticsearch is reachable (HTTP 401)

ES_PATH_CONF: /etc/elasticsearch/user-universe-main-dev/node-01
HTTP URL:    https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

Running HTTP reset (online, --batch)…
Note: Using HTTPS - tools will read SSL config from elasticsearch.yml
Command output:
Password for the [elastic] user successfully reset.
New value: 9*SLrPDh01M+gJSVeRxz
Exit status: 0
✓ HTTP reset succeeded for elastic
Storing credentials in AWS Secrets Manager: fastorder/search/user/universe/main/dev/elasticsearch/node-01
ℹ️  Setting Elasticsearch credentials in vault: fastorder/search/user/universe/main/dev/elasticsearch/node-01
ℹ️  Setting secret in AWS Secrets Manager: fastorder/search/user/universe/main/dev/elasticsearch/node-01
✅ Secret updated: fastorder/search/user/universe/main/dev/elasticsearch/node-01
✅ Elasticsearch credentials set in vault: fastorder/search/user/universe/main/dev/elasticsearch/node-01
✓ Password stored in AWS Secrets Manager: fastorder/search/user/universe/main/dev/elasticsearch/node-01
✓ Cache cleared for: fastorder/search/user/universe/main/dev/elasticsearch/node-01

✓ Done. Password stored in AWS Secrets Manager: fastorder/search/user/universe/main/dev/elasticsearch/node-01

Usage Examples:
  # Retrieve password using AWS CLI
  aws secretsmanager get-secret-value --secret-id fastorder/search/user/universe/main/dev/elasticsearch/node-01 --region ${AWS_REGION:-me-central-1}

  # Using fastctl
  fastctl secrets get fastorder/search/user/universe/main/dev/elasticsearch/node-01

  # Test connection
  curl -u elastic:$(fastctl secrets get fastorder/search/user/universe/main/dev/elasticsearch/node-01 --field password) https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health
✓ Retrieved password from AWS Secrets Manager
[INFO] Retrying authentication with new password...
[INFO] Using client cert/key for mTLS: /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.key
[INFO]   ⏳ waiting for secure cluster health (require 200) at https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200 (timeout 300s)...
[ OK ] Cluster health OK: green

==================================================================
[ OK ] All services restarted successfully!
[ OK ] Cluster is healthy, HTTPS-secure, and responding with 200
[INFO] Environment: user-universe-main-dev
[INFO] Services: 1
[INFO] Endpoint: https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === Manual verification (copy/paste) ===
curl -u 'elastic:9*SLrPDh01M+gJSVeRxz' \
  --cacert '/etc/elasticsearch/user-universe-main-dev/node-01/certs/http_ca.crt' \
  --cert   '/etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.crt' \
  --key    '/etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.key' \
  'https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty'

[INFO] === Quick checks ===
curl -u 'elastic:9*SLrPDh01M+gJSVeRxz' --cacert '/etc/elasticsearch/user-universe-main-dev/node-01/certs/http_ca.crt' --cert '/etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.crt' --key '/etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.key' https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cat/nodes?v
curl -u 'elastic:9*SLrPDh01M+gJSVeRxz' --cacert '/etc/elasticsearch/user-universe-main-dev/node-01/certs/http_ca.crt' --cert '/etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.crt' --key '/etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.key' https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/
Executing: steps/05-test-elastic.sh
==================================================================
STEP 5: Test Elasticsearch Cluster
==================================================================
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: user-universe-main-dev
Nodes: 1
Node: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.219)
Port: 9200 (default port)

[INFO] Using centralized test suite: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/02-make-https/lib/elasticsearch-test-suite.sh
[INFO] Executing centralized test suite with args: -v -t all --env user-universe-main-dev -u elastic
[INFO] Using CURRENT_ENV_ID from environment: user-universe-main-dev
[INFO] Loaded from topology.json: user-universe-main-dev
[2026-01-12 20:41:35] Loaded environment: user-universe-main-dev
[2026-01-12 20:41:35] Service: user, Zone: universe, Branch: main, Env: dev
[2026-01-12 20:41:35] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-12 20:41:35] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-12 20:41:35] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
ℹ Using CURRENT_ENV_ID from environment: user-universe-main-dev
ℹ Loaded from topology.json: user-universe-main-dev
[2026-01-12 20:41:36] Loaded environment: user-universe-main-dev
[2026-01-12 20:41:36] Service: user, Zone: universe, Branch: main, Env: dev
[2026-01-12 20:41:36] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-12 20:41:36] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-12 20:41:36] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
ℹ Project root: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch
ℹ Environment:  user-universe-main-dev
ℹ Nodes count:  1
ℹ Endpoint:      https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
ℹ Using CA:       /etc/elasticsearch/user-universe-main-dev/node-01/certs/http_ca.crt
ℹ Using mTLS:     /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.key
╔════════════════════════════════════════════╗
║    Elasticsearch Centralized Test Suite    ║
╚════════════════════════════════════════════╝

=== Authentication Test ===
✓ Loaded credentials for user elastic from AWS Secrets Manager
Curl (local): curl --cacert /etc/elasticsearch/user-universe-main-dev/node-01/certs/http_ca.crt --cert /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.crt --key /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.key  -u 'elastic:********' 'https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty'
✓ Local authentication successful (HTTP 200).
{
  "cluster_name" : "fastorder-user-universe-main-dev",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 3,
  "active_shards" : 3,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Executing: steps/06-final-testing.sh
==================================================================
STEP 6: Final Testing and Verification
==================================================================
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: user-universe-main-dev
Nodes: 1
Node: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.219)
Port: 9200 (default port)

[INFO] Using centralized test suite: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/02-make-https/lib/elasticsearch-test-suite.sh
[INFO] Using CURRENT_ENV_ID from environment: user-universe-main-dev
[INFO] Loaded from topology.json: user-universe-main-dev
[2026-01-12 20:41:38] Loaded environment: user-universe-main-dev
[2026-01-12 20:41:38] Service: user, Zone: universe, Branch: main, Env: dev
[2026-01-12 20:41:38] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-12 20:41:38] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-12 20:41:38] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
ℹ Using CURRENT_ENV_ID from environment: user-universe-main-dev
ℹ Loaded from topology.json: user-universe-main-dev
[2026-01-12 20:41:39] Loaded environment: user-universe-main-dev
[2026-01-12 20:41:39] Service: user, Zone: universe, Branch: main, Env: dev
[2026-01-12 20:41:39] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-12 20:41:39] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-12 20:41:39] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
ℹ Project root: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch
ℹ Environment:  user-universe-main-dev
ℹ Nodes count:  1
ℹ Endpoint:      https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
ℹ Using CA:       /etc/elasticsearch/user-universe-main-dev/node-01/certs/http_ca.crt
ℹ Using mTLS:     /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.key
╔════════════════════════════════════════════╗
║    Elasticsearch Centralized Test Suite    ║
╚════════════════════════════════════════════╝

=== Authentication Test ===
✓ Loaded credentials for user elastic from AWS Secrets Manager
Curl (local): curl --cacert /etc/elasticsearch/user-universe-main-dev/node-01/certs/http_ca.crt --cert /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.crt --key /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.key  -u 'elastic:********' 'https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty'
✓ Local authentication successful (HTTP 200).
{
  "cluster_name" : "fastorder-user-universe-main-dev",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 3,
  "active_shards" : 3,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Executing: steps/07-set-passwords.sh
==================================================================
STEP 7: Setting cluster passwords (bootstrap via alias)
==================================================================
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Using HTTPS with CA: /etc/elasticsearch/user-universe-main-dev/node-01/certs/http_ca.crt (host: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com)
[INFO] Using centralized password setter: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/02-make-https/steps/../lib/elasticsearch-set-password.sh
[ OK ] Elastic password already valid (HTTP 200) via search-user-universe-main-dev-elasticsearch-node-01.fastorder.com; nothing to do.
Executing: steps/08-create-app-user.sh
==================================================================
STEP 8: Create Application User and Roles (cluster-scoped)
==================================================================
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: user-universe-main-dev
Nodes: 1

[INFO] Using HTTPS with CA: /etc/elasticsearch/user-universe-main-dev/node-01/certs/http_ca.crt (host: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com)
[ OK ] Retrieved elastic password from Vault (cluster scope).
[INFO] Configuration:
[INFO]   App User         : app_user
[INFO]   Read-only Role   : app_ro
[INFO]   Read-write Role  : app_rw
[INFO]   Index Patterns   : app-*,cdc-*,user_universe_*,*_account_router
[INFO]   Endpoint         : https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] Creating read-only role: app_ro
[ OK ] ✓ Role app_ro ensured
[INFO] Creating read-write role: app_rw
[ OK ] ✓ Role app_rw ensured
[INFO] Creating/Updating application user: app_user
[ OK ] ✓ User app_user ensured
ℹ️  Setting Elasticsearch credentials in vault: fastorder/search/user/universe/main/dev/elasticsearch/node-01/app_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/search/user/universe/main/dev/elasticsearch/node-01/app_user
✅ Secret updated: fastorder/search/user/universe/main/dev/elasticsearch/node-01/app_user
✅ Elasticsearch credentials set in vault: fastorder/search/user/universe/main/dev/elasticsearch/node-01/app_user
[ OK ] ✓ Stored app_user password under 'node-01/app_user'
ℹ️  Setting Elasticsearch credentials in vault: fastorder/search/user/universe/main/dev/elasticsearch/cluster/app_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/search/user/universe/main/dev/elasticsearch/cluster/app_user
✅ Secret updated: fastorder/search/user/universe/main/dev/elasticsearch/cluster/app_user
✅ Elasticsearch credentials set in vault: fastorder/search/user/universe/main/dev/elasticsearch/cluster/app_user
[ OK ] ✓ Stored app_user password under 'cluster/app_user'
[INFO] Testing authentication for app_user...
[ OK ] ✓ Authentication test passed for app_user

[ OK ] ✓ Application user and roles created successfully!
[INFO] User    : app_user
[INFO] Roles   : app_ro, app_rw
[INFO] Patterns: app-*,cdc-*,user_universe_*,*_account_router
[INFO] Endpoint: https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
Executing: steps/09-config.sh
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
Environment: user-universe-main-dev
Nodes: 1
Node: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.219)
Port: 9200

✓ Auto mode: Cloud IMDS detected → MODE=role
[INFO] Mode: role

[INFO] AWS Region: me-central-1
[INFO] MODE=role → will purge any static S3 keys from each node keystore

[2026-01-12 20:43:00 UTC] USER=www-data EUID=0 PID=705061 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-universe-main-dev/node-01
[INFO] • node-01 keystore cleared (role-based auth)
[2026-01-12 20:43:09 UTC] USER=www-data EUID=0 PID=705383 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-12 20:43:09 UTC] USER=www-data EUID=0 PID=705431 ACTION=passthru ARGS=systemctl restart elasticsearch@user-universe-main-dev-node-01.service
✓ ✓ restarted elasticsearch@user-universe-main-dev-node-01.service

⏳ Waiting for HTTPS readiness on https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Waiting HTTP readiness at https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/ (200/401/302)…
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
[OK] Ready: 401
⏳ Waiting for cluster health (green|yellow)
[INFO] Waiting health (green|yellow) at https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health…
[OK] 401 pre-auth received; security enabled.
✓ ✓ user-universe-main-dev is responding via search-user-universe-main-dev-elasticsearch-node-01.fastorder.com

✓ ✓ AWS S3 configuration completed for environment: user-universe-main-dev (1 nodes)
[INFO] Mode: role
[INFO] Region: me-central-1
Executing: steps/0ld-03-http-ssl.sh
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
==================================================================
STEP 5: HTTP SSL Configuration (Optional)
==================================================================
Environment: user-universe-main-dev
Nodes: 1
Node: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.219)
Port: 9200 (default port)

[ OK ] 🚀 Auto mode/Default installation: Selecting Direct HTTPS configuration (option 1)

[ OK ] Configuring Direct HTTPS (Elasticsearch native SSL)...
──────────────────────────────────────────────────────────
[INFO] Environment: user-universe-main-dev (1 nodes)
[INFO] Node: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.219)
[INFO] Port: 9200 (default port)
==================================================================
Direct HTTPS Configuration (native TLS, PEM)
==================================================================
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
Environment: user-universe-main-dev
Nodes: 1
Node: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.219)
Port: 9200 (default port)
Domain: user-universe-main-dev.fastorder.com

[INFO] === Single-Node Direct HTTPS Setup ===
🔐 Generated PKCS12 password: 9ZFo0PnN... (32 chars)
[INFO] Using first node configuration: /etc/elasticsearch/user-universe-main-dev/node-01
[2026-01-12 20:43:48 UTC] USER=www-data EUID=0 PID=706842 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-universe-main-dev/node-01
[2026-01-12 20:43:48 UTC] USER=www-data EUID=0 PID=706851 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-universe-main-dev/node-01
[INFO] Checking prerequisites...
[ OK ] ✓ Prerequisites verified
[INFO] Setting up temporary directories...
[2026-01-12 20:43:48 UTC] USER=www-data EUID=0 PID=706862 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out /etc/elasticsearch/user-universe-main-dev/node-01/http-certs
[2026-01-12 20:43:48 UTC] USER=www-data EUID=0 PID=706880 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-universe-main-dev
[2026-01-12 20:43:48 UTC] USER=www-data EUID=0 PID=706889 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-universe-main-dev/node-01
[2026-01-12 20:43:48 UTC] USER=www-data EUID=0 PID=706898 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-universe-main-dev/node-01/http-certs
[2026-01-12 20:43:48 UTC] USER=www-data EUID=0 PID=706907 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out
[ OK ] ✓ Directories created

[INFO] Building certificate instances configuration...
[INFO] Certificate instances configuration:
instances:
  - name: "user-universe-main-dev-http"
    dns:  [ "localhost", "web-03", "search-user-universe-main-dev-elasticsearch-node-01.fastorder.com", "user-universe-main-dev-node-01.fastorder.com", "search-user-universe-main-dev-elasticsearch-coordinator.fastorder.com", "search-user-universe-main-dev.fastorder.com", "user-universe-main-dev-node-01.local" ]
    ip:   [ "10.100.1.219", "127.0.0.1", "::1" ]
[ OK ] ✓ Instances configuration created

[INFO] Generating HTTP Certificate Authority...
[2026-01-12 20:43:49 UTC] USER=www-data EUID=0 PID=706926 ACTION=fsop ARGS=rm -f /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/http-ca.zip /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out/http-certs.zip
[2026-01-12 20:43:49 UTC] USER=www-data EUID=0 PID=706935 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-universe-main-dev/node-01/http-certs
[2026-01-12 20:43:49 UTC] USER=www-data EUID=0 PID=706944 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-universe-main-dev/node-01/http-certs
[2026-01-12 20:43:52 UTC] USER=www-data EUID=0 PID=707056 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/http-ca.zip
Archive:  /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/http-ca.zip
  inflating: /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/ca/ca.crt  
  inflating: /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/ca/ca.key  
[ OK ] ✓ HTTP CA generated successfully

[INFO] Generating per-node HTTP certificates...
Archive:  /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out/http-certs.zip
  inflating: /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out/http/user-universe-main-dev-http/user-universe-main-dev-http.crt  
  inflating: /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out/http/user-universe-main-dev-http/user-universe-main-dev-http.key  
[ OK ] ✓ HTTP certificates generated successfully

[INFO] Installing certificates and configuring services...
[INFO] Configuring main service for single-node HTTPS...
[2026-01-12 20:43:56 UTC] USER=www-data EUID=0 PID=707218 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-universe-main-dev/node-01/certs
[2026-01-12 20:43:56 UTC] USER=www-data EUID=0 PID=707230 ACTION=fsop ARGS=cp /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out/http/user-universe-main-dev-http/user-universe-main-dev-http.crt /etc/elasticsearch/user-universe-main-dev/node-01/certs/http.crt
[2026-01-12 20:43:56 UTC] USER=www-data EUID=0 PID=707239 ACTION=fsop ARGS=cp /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/out/http/user-universe-main-dev-http/user-universe-main-dev-http.key /etc/elasticsearch/user-universe-main-dev/node-01/certs/http.key
[2026-01-12 20:43:56 UTC] USER=www-data EUID=0 PID=707261 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/user-universe-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-01-12 20:43:56 UTC] USER=www-data EUID=0 PID=707271 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-universe-main-dev/node-01/certs
[2026-01-12 20:43:56 UTC] USER=www-data EUID=0 PID=707289 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/user-universe-main-dev/node-01/certs/http.crt /etc/elasticsearch/user-universe-main-dev/node-01/certs/http_ca.crt /etc/elasticsearch/user-universe-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-01-12 20:43:56 UTC] USER=www-data EUID=0 PID=707307 ACTION=fsop ARGS=sed -i /^# --- BEGIN direct HTTPS (managed, PEM) ---$/,/^# --- END direct HTTPS (managed, PEM) ---$/d /etc/elasticsearch/user-universe-main-dev/node-01/elasticsearch.yml
[2026-01-12 20:43:56 UTC] USER=www-data EUID=0 PID=707316 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/user-universe-main-dev/node-01/certs/fastorder_ra_root.crt
YAML: /etc/elasticsearch/user-universe-main-dev/node-01/elasticsearch.yml
[ OK ]   ✓ Main service configured with HTTPS
[2026-01-12 20:43:56 UTC] USER=www-data EUID=0 PID=707335 ACTION=fsop ARGS=rm -rf /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients
Archive:  /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client.zip
   creating: /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/
  inflating: /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.crt  
  inflating: /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.key  
[INFO] Creating P12 keystore for es-client...
[2026-01-12 20:44:00 UTC] USER=www-data EUID=0 PID=707468 ACTION=fsop ARGS=mv /tmp/es-client-706785.p12 /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[ OK ] ✓ Created P12 keystore: /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-01-12 20:44:00 UTC] USER=www-data EUID=0 PID=707477 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients
[2026-01-12 20:44:00 UTC] USER=www-data EUID=0 PID=707486 ACTION=fsop ARGS=chmod 640 /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.key
[2026-01-12 20:44:00 UTC] USER=www-data EUID=0 PID=707495 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[INFO] Saving keystore passwords to secrets vault...
🔑 Configuring AWS credentials for secrets vault...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[INFO] 🔐 Vaulting search passwords to remote backend...
✅ Passwords vaulted to remote backend
✓ Keystore passwords saved to secrets vault: search/user-universe-main-dev/keystore-passwords

[INFO] === Installing CA Certificate for Users ===
[INFO] HOME not set, skipping user CA installation

✓ Direct HTTPS configuration completed for environment: user-universe-main-dev
[INFO] All services now serve HTTPS using PEM certificates
[INFO] Network binding: 10.100.1.219
[INFO] HTTPS endpoint: https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === Certificate Summary ===
CA Certificate: /etc/elasticsearch/user-universe-main-dev/node-01/http-certs/ca/ca.crt
Certificate Directory: /etc/elasticsearch/user-universe-main-dev/node-01/http-certs
Main service certificates: /etc/elasticsearch/user-universe-main-dev/node-01/certs/

[INFO] === Next Steps ===
1. Restart Elasticsearch services to apply HTTPS configuration
2. Test HTTPS connectivity: curl https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
3. Verify certificates: openssl s_client -connect search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === SSL Certificate Information ===
• CA certificate installed for user - curl will work without -k flag
• Each certificate includes node-specific domain, VM IP, and localhost in Subject Alternative Names
• Certificates are valid for 3 years from generation date

[WARNING] Important: You'll need to restart Elasticsearch services for HTTPS to take effect
[ OK ] ✓ Direct HTTPS configuration completed successfully

[ OK ] ==================================================================
[ OK ] HTTP SSL Configuration Complete
[ OK ] ==================================================================
[INFO] Environment: user-universe-main-dev
[INFO] Nodes: 1
[INFO] Configuration applied to port: 9200 (default port for all nodes)

[INFO] === Next Steps ===
1. Verify Elasticsearch is running: systemctl status elasticsearch@user-universe-main-dev-node-01.service
2. Test cluster health: curl https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health
3. Check SSL certificate: openssl s_client -connect search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
=== HTTPS Setup completed successfully! ===
Environment:  (1 nodes)
Domain: .fastorder.com
HTTPS endpoint: https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
Node IP: 10.100.1.219

✓ Step 2 completed successfully!

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 3: Executing Create Index Llm
Folder: 03-create-index-llm
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

==================================================================
Elasticsearch LLM/Semantic Search Setup
==================================================================
[INFO] Using web-provided environment: user-universe-main-dev
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: user-universe-main-dev
Service    : user
🔍 Checking Elasticsearch availability…
✅ Elasticsearch is accessible at https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

=== Phase 1: Common steps under /data/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps ===
   (no numbered steps in: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps)
=== Phase 2: Service-scoped steps for 'user' under /data/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/user ===
📚 Detected features: contracts

── Feature: contracts
▶️  Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/user/contracts/00-create-ingest-pipeline.sh
==================================================================
STEP 0: Create Ingest Pipeline (User Contracts)
==================================================================
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Creating ingest pipeline: user_universe_main_dev_user_contracts_pipeline
[OK]   Ingest pipeline created: user_universe_main_dev_user_contracts_pipeline
[INFO] Testing pipeline with sample document...
[OK]   Pipeline simulation completed

==================================================================
Ingest Pipeline Configuration Complete
==================================================================
  Pipeline: user_universe_main_dev_user_contracts_pipeline
  ES URL: https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_ingest/pipeline/user_universe_main_dev_user_contracts_pipeline

  OPERATIONS:
    - Default values for status fields
    - Lowercase normalization (username)
    - Uppercase normalization (country_code)
    - Timestamp parsing (created_at, updated_at, dates)
    - Safety net removal of raw PII fields

  NOT DONE (by design):
    - Email/phone hashing (done at PostgreSQL level)
    - PII transformation (should never reach this pipeline)
==================================================================
✅ 00-create-ingest-pipeline.sh completed

▶️  Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/user/contracts/01-create-model-and-pipeline.sh
==================================================================
STEP 1: Create Model and Ingest Pipeline (User Contracts)
==================================================================
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] ES URL: https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Endpoint ID (ES): user-contracts-embedding-001
[INFO] Provider model: text-embedding-3-large
[INFO] Pipeline ID: user-contracts-embed-pipeline-001
[INFO] Checking authentication identity…
{
"username":"elastic","roles":["superuser"],"full_name":null,"email":null,"metadata":{"_reserved":true},"enabled":true,"authentication_realm":{"name":"reserved","type":"reserved"},"lookup_realm":{"name":"reserved","type":"reserved"},"authentication_type":"realm"
}
[INFO] Checking Elasticsearch license…
[INFO] License type: unknown
[WARN] Inference API requires Enterprise/Platinum license (found: unknown)
[WARN] Skipping inference endpoint and pipeline creation
[OK]   Setup completed (inference features skipped due to license)
✅ 01-create-model-and-pipeline.sh completed

▶️  Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/user/contracts/02-create-index.sh
==================================================================
STEP 2: Create Minimal User Contracts Index (ILM bootstrap)
==================================================================
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] [cluster] Ensure disk watermarks permit allocation
[OK]   Cluster watermarks set/confirmed.
[INFO] [create] Create/Update ILM policy: user-user-contracts-ilm
[OK]   ILM policy ready.
[INFO] [create] Create/Update index template: user_universe_main_dev_user_contracts_template
[OK]   Index template ready.
[INFO] [check] Concrete index: user_universe_main_dev_user_contracts-000001
[INFO] [create] Create first index + attach write alias: user_universe_main_dev_user_contracts-000001
[OK]   Created user_universe_main_dev_user_contracts-000001 with write alias user_universe_main_dev_user_contracts.
[INFO] [verify] Wait for index to be at least YELLOW
[OK]   Cluster health OK for user_universe_main_dev_user_contracts-000001.
[INFO] [verify] Alias points to a concrete write index
[OK]   Alias verification passed.
[INFO] [explain] ILM status
{
  "indices" : {
    "user_universe_main_dev_user_contracts-000001" : {
      "index" : "user_universe_main_dev_user_contracts-000001",
      "managed" : true,
      "policy" : "user-user-contracts-ilm",
      "index_creation_date_millis" : 1768250654898,
      "time_since_index_creation" : "357ms",
      "lifecycle_date_millis" : 1768250654898,
      "age" : "357ms",
      "phase" : "hot",
      "phase_time_millis" : 1768250655019,
      "action" : "rollover",
      "action_time_millis" : 1768250655019,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1768250655019,
      "phase_execution" : {
        "policy" : "user-user-contracts-ilm",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_age" : "30d",
              "max_primary_shard_docs" : 200000000,
              "min_docs" : 1,
              "max_primary_shard_size" : "5gb"
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1768250654372
      },
      "skip" : false
    }
  }
}

[OK]   Minimal User Contracts Index bootstrap complete.
   Index (concrete): user_universe_main_dev_user_contracts-000001
   Alias (stable)  : user_universe_main_dev_user_contracts  (is_write_index=true)
   ILM policy      : user-user-contracts-ilm
   Default pipeline: user-contracts-embed-pipeline-001
   Vectors         : dense_vector dims=1536 (KNN cosine)

==================================================================
PRIVACY-BY-DESIGN: Minimal User Index Compliance
==================================================================
   INDEXED: user_id, tenant_id, home_region, status, username, display_name
            email_hash, phone_hash, country_code, region_code, tags, segments
            contract_id, contract_type, contract_status, dates
   EXCLUDED: gender, age, DOB, national_id, exact_address, payment_data
   PRINCIPLE: Route to region + safe display snippet, load full details from Postgres
==================================================================
✅ 02-create-index.sh completed

▶️  Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/user/contracts/03-llm.sh
==================================================================
STEP 3: LLM Semantic Search Configuration (User Contracts)
==================================================================
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Checking Elasticsearch license for semantic search...
[INFO] License type: unknown
[WARN] Semantic search features require Enterprise/Platinum license (found: unknown)
[WARN] Text-based search will still work, but KNN/vector search is unavailable
[INFO] [verify] Checking inference endpoint: user-contracts-embedding-001
[WARN] Inference endpoint user-contracts-embedding-001 not found (HTTP 404)
[WARN] Run 01-create-model-and-pipeline.sh first to create the endpoint
[WARN] Semantic search will fall back to text-based search
[INFO] [create] Creating search template for user contracts...
[WARN] Search template creation returned HTTP 400. Response:
HTTP/1.1 400 Bad Request
X-elastic-product: Elasticsearch
content-type: application/json
content-length: 1847

{"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[40:13] [stored script source] failed to parse field [source]"}],"type":"x_content_parse_exception","reason":"[40:13] [stored script source] failed to parse field [source]","caused_by":{"type":"unchecked_i_o_exception","reason":"com.fasterxml.jackson.core.JsonParseException: Unexpected character ('{' (code 123)): was expecting double-quote to start field name\n at [Source: (byte[])\"{\n  \"script\": {\n    \"lang\": \"mustache\",\n    \"source\": {\n      \"size\": \"{{size}}{{^size}}10{{/size}}\",\n      \"query\": {\n        \"bool\": {\n          \"should\": [\n            {\n              \"multi_match\": {\n                \"query\": \"{{query}}\",\n                \"fields\": [\n                  \"username^3\",\n                  \"display_name^2\",\n                  \"contract_summary_en\",\n                  \"contract_summary_ar\"\n                ],\n                \"type\": \"best_fields\",\n                \"fuzzin\"[truncated 1411 bytes]; line: 40, column: 14]","caused_by":{"type":"json_parse_exception","reason":"Unexpected character ('{' (code 123)): was expecting double-quote to start field name\n at [Source: (byte[])\"{\n  \"script\": {\n    \"lang\": \"mustache\",\n    \"source\": {\n      \"size\": \"{{size}}{{^size}}10{{/size}}\",\n      \"query\": {\n        \"bool\": {\n          \"should\": [\n            {\n              \"multi_match\": {\n                \"query\": \"{{query}}\",\n                \"fields\": [\n                  \"username^3\",\n                  \"display_name^2\",\n                  \"contract_summary_en\",\n                  \"contract_summary_ar\"\n                ],\n                \"type\": \"best_fields\",\n                \"fuzzin\"[truncated 1411 bytes]; line: 40, column: 14]"}}},"status":400}
[INFO] [create] Creating KNN search template for semantic similarity...
[WARN] KNN template creation returned HTTP 400. Response:
HTTP/1.1 400 Bad Request
X-elastic-product: Elasticsearch
content-type: application/json
content-length: 1831

{"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[18:9] [stored script source] failed to parse field [source]"}],"type":"x_content_parse_exception","reason":"[18:9] [stored script source] failed to parse field [source]","caused_by":{"type":"unchecked_i_o_exception","reason":"com.fasterxml.jackson.core.JsonParseException: Unexpected character ('{' (code 123)): was expecting double-quote to start field name\n at [Source: (byte[])\"{\n  \"script\": {\n    \"lang\": \"mustache\",\n    \"source\": {\n      \"size\": \"{{size}}{{^size}}10{{/size}}\",\n      \"knn\": {\n        \"field\": \"{{vector_field}}{{^vector_field}}embedding_en{{/vector_field}}\",\n        \"query_vector_builder\": {\n          \"text_embedding\": {\n            \"model_id\": \"user-contracts-embedding-001\",\n            \"model_text\": \"{{query}}\"\n          }\n        },\n        \"k\": \"{{k}}{{^k}}10{{/k}}\",\n        \"num_candidates\": \"{{num_candidates}}{{^num_candidates}}100{{/num_candidate\"[truncated 662 bytes]; line: 18, column: 10]","caused_by":{"type":"json_parse_exception","reason":"Unexpected character ('{' (code 123)): was expecting double-quote to start field name\n at [Source: (byte[])\"{\n  \"script\": {\n    \"lang\": \"mustache\",\n    \"source\": {\n      \"size\": \"{{size}}{{^size}}10{{/size}}\",\n      \"knn\": {\n        \"field\": \"{{vector_field}}{{^vector_field}}embedding_en{{/vector_field}}\",\n        \"query_vector_builder\": {\n          \"text_embedding\": {\n            \"model_id\": \"user-contracts-embedding-001\",\n            \"model_text\": \"{{query}}\"\n          }\n        },\n        \"k\": \"{{k}}{{^k}}10{{/k}}\",\n        \"num_candidates\": \"{{num_candidates}}{{^num_candidates}}100{{/num_candidate\"[truncated 662 bytes]; line: 18, column: 10]"}}},"status":400}
[INFO] [create] Creating user lookup template for exact matches...
[WARN] Lookup template creation returned HTTP 400. Response:
HTTP/1.1 400 Bad Request
X-elastic-product: Elasticsearch
content-type: application/json
content-length: 1831

{"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[9:13] [stored script source] failed to parse field [source]"}],"type":"x_content_parse_exception","reason":"[9:13] [stored script source] failed to parse field [source]","caused_by":{"type":"unchecked_i_o_exception","reason":"com.fasterxml.jackson.core.JsonParseException: Unexpected character ('{' (code 123)): was expecting double-quote to start field name\n at [Source: (byte[])\"{\n  \"script\": {\n    \"lang\": \"mustache\",\n    \"source\": {\n      \"size\": 1,\n      \"query\": {\n        \"bool\": {\n          \"must\": [\n            {{#email_hash}}\n            { \"term\": { \"email_hash\": \"{{email_hash}}\" } }\n            {{/email_hash}}\n            {{#phone_hash}}\n            { \"term\": { \"phone_hash\": \"{{phone_hash}}\" } }\n            {{/phone_hash}}\n            {{#user_id}}\n            { \"term\": { \"user_id\": \"{{user_id}}\" } }\n            {{/user_id}}\n            {{#username}}\n            {\"[truncated 530 bytes]; line: 9, column: 14]","caused_by":{"type":"json_parse_exception","reason":"Unexpected character ('{' (code 123)): was expecting double-quote to start field name\n at [Source: (byte[])\"{\n  \"script\": {\n    \"lang\": \"mustache\",\n    \"source\": {\n      \"size\": 1,\n      \"query\": {\n        \"bool\": {\n          \"must\": [\n            {{#email_hash}}\n            { \"term\": { \"email_hash\": \"{{email_hash}}\" } }\n            {{/email_hash}}\n            {{#phone_hash}}\n            { \"term\": { \"phone_hash\": \"{{phone_hash}}\" } }\n            {{/phone_hash}}\n            {{#user_id}}\n            { \"term\": { \"user_id\": \"{{user_id}}\" } }\n            {{/user_id}}\n            {{#username}}\n            {\"[truncated 530 bytes]; line: 9, column: 14]"}}},"status":400}

[OK]   LLM configuration complete for User Contracts index.

Available search templates:
   1. user_universe_main_dev_user_contracts_search      - Text + filter search
   2. user_universe_main_dev_user_contracts_knn_search  - KNN vector similarity search
   3. user_universe_main_dev_user_contracts_lookup  - Exact match lookup (by hash)

Usage examples:
   Text search:
     POST https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/user_universe_main_dev_user_contracts/_search/template
     { "id": "user_universe_main_dev_user_contracts_search", "params": { "query": "service agreement", "tenant_id": "t1" } }

   KNN semantic search:
     POST https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/user_universe_main_dev_user_contracts/_search/template
     { "id": "user_universe_main_dev_user_contracts_knn_search", "params": { "query": "find contracts about data privacy" } }

   User lookup (by hashed email):
     POST https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/user_universe_main_dev_user_contracts/_search/template
     { "id": "user_universe_main_dev_user_contracts_lookup", "params": { "email_hash": "<sha256_hash>" } }
✅ 03-llm.sh completed

▶️  Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/user/contracts/04-index-sample-data.sh
==================================================================
STEP 4: Index Sample Data (User Contracts - Minimal Index)
==================================================================
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[WARN] Pipeline 'user-contracts-embed-pipeline-001' not found (HTTP 404); proceeding without it.
[INFO] [bulk] Index seed documents -> user_universe_main_dev_user_contracts
[INFO] PRIVACY: Using hashed email/phone, coarse location, no sensitive PII
[WARN] Bulk completed with item-level errors. Showing first 50 lines:
{"errors":true,"took":0,"ingest_took":0,"items":[{"index":{"_index":"user_universe_main_dev_user_contracts","_id":"auto-generated","status":400,"error":{"type":"illegal_argument_exception","reason":"pipeline with id [user-contracts-embed-pipeline-001] does not exist"}}},{"index":{"_index":"user_universe_main_dev_user_contracts","_id":"auto-generated","status":400,"error":{"type":"illegal_argument_exception","reason":"pipeline with id [user-contracts-embed-pipeline-001] does not exist"}}},{"index":{"_index":"user_universe_main_dev_user_contracts","_id":"auto-generated","status":400,"error":{"type":"illegal_argument_exception","reason":"pipeline with id [user-contracts-embed-pipeline-001] does not exist"}}},{"index":{"_index":"user_universe_main_dev_user_contracts","_id":"auto-generated","status":400,"error":{"type":"illegal_argument_exception","reason":"pipeline with id [user-contracts-embed-pipeline-001] does not exist"}}},{"index":{"_index":"user_universe_main_dev_user_contracts","_id":"auto-generated","status":400,"error":{"type":"illegal_argument_exception","reason":"pipeline with id [user-contracts-embed-pipeline-001] does not exist"}}}]}[summary] items=5 errors=5
[INFO] [verify] Search a sample term: 'service agreement'
  {
    "took" : 75,
    "timed_out" : false,
    "_shards" : {
      "total" : 1,
      "successful" : 1,
      "skipped" : 0,
      "failed" : 0
    },
    "hits" : {
      "total" : {
        "value" : 0,
        "relation" : "eq"
      },
      "max_score" : null,
      "hits" : [ ]
    }
  }

[OK]   Sample data indexing step completed.

==================================================================
PRIVACY COMPLIANCE: Sample Data Structure
==================================================================
   INDEXED (minimal):
     - Identity: user_id, tenant_id, home_region, home_env, status
     - Search: username, display_name, email_hash, phone_hash
     - Location: country_code, region_code (coarse only)
     - Metadata: tags, segments, flags, external_refs
     - Contract: contract_id, type, status, dates, summary

   EXCLUDED (by design):
     - Gender, Age, Date of Birth
     - National ID, Passport Number
     - Exact Address, Precise GPS Coordinates
     - Payment/Financial Data

   PRINCIPLE: Universal index for routing + lookup only.
              Full user details loaded from zonal Postgres.
==================================================================
✅ 04-index-sample-data.sh completed

▶️  Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/user/contracts/05-create-cdc-index.sh
==================================================================
STEP 5: Create CDC User Contracts Index (for dashboard visibility)
==================================================================
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Creating CDC index: user_universe_main_dev_user_contracts_cdc
[INFO] Index does not exist (status 404), creating...
[OK]   CDC index created successfully
   Index: user_universe_main_dev_user_contracts_cdc
   URL: https://search-user-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/user_universe_main_dev_user_contracts_cdc

==================================================================
PRIVACY-BY-DESIGN: CDC Index Structure
==================================================================
   INDEXED (in 'after' object):
     - Identity: user_id, tenant_id, home_region, home_env, status
     - Search: username, display_name (email_hash/phone_hash stored only)
     - Location: country_code, region_code (coarse)
     - Metadata: tags, segments, flags, external_refs
     - Contract: contract_id, type, status, dates

   STORED BUT NOT INDEXED (in 'before' object):
     - user_id, tenant_id, status, contract_status (for audit trail)

   DISABLED (not stored):
     - source (Debezium metadata, not needed for search)

   EXCLUDED BY DESIGN:
     - Gender, Age, DOB, National ID
     - Exact Address, Precise GPS
     - Payment/Financial Data
==================================================================

--- Dashboard can now list this index before CDC pipeline writes data
--- ES Sink connector will write CDC data to this index automatically
✅ 05-create-cdc-index.sh completed

=== Phase 3: Optional search smoke tests ===
   (semantic search test script not found: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/search-semantic.sh)
   (hybrid search test script not found: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/hybrid-search.sh)
==================================================================
🎉 LLM/Semantic Search setup completed successfully!
==================================================================

Available commands:
  • Test semantic search:
    bash steps/search-semantic.sh en "password policy"
    bash steps/search-semantic.sh ar "كلمة المرور"

  • Test hybrid search:
    bash steps/hybrid-search.sh en "user authentication"
    bash steps/hybrid-search.sh ar "مصادقة المستخدم"

Alias   : user_universe_main_dev_account_router
Index   : user_universe_main_dev_account_router-000001
ILM     : user-account-router-ilm
Model   : user-text-embedding-001
Pipeline: user-embed-pipeline-001
==================================================================

✓ Step 3 completed successfully!

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 4: Executing Monitoring Setup
Folder: 10-monitoring-setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using web-provided environment: user-universe-main-dev
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Elasticsearch Monitoring Integration for user-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-user-universe-main-dev
[OK]   Observability cell endpoints registered for user-universe-main-dev
[OK]   ✓ Observability cell is ready

[INFO] 2️⃣ Discovering Elasticsearch configuration...
[OK]   ✓ Found Elasticsearch at 10.100.1.219:9200

[INFO] 3️⃣ Setting up elasticsearch_exporter integration...
[INFO] Using elasticsearch_exporter port: 9114
[INFO] SSL certificates configured for elasticsearch_exporter:
[INFO]   CA cert: /etc/elasticsearch/user-universe-main-dev/node-01/certs/http_ca.crt
[INFO]   Client cert: /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.crt
[INFO]   Client key: /etc/elasticsearch/user-universe-main-dev/node-01/certs/clients/es-client/es-client.key
[INFO] Checking observability cell readiness: obs-user-universe-main-dev
[OK]   Observability cell endpoints registered for user-universe-main-dev
[INFO] Setting up elasticsearch_exporter for user-universe-main-dev
[INFO] Elasticsearch exporter will bind to: 10.100.1.219:9114
[2026-01-12 20:44:22 UTC] USER=www-data EUID=0 PID=709073 ACTION=passthru ARGS=mv /tmp/elasticsearch_exporter-user-universe-main-dev.service /etc/systemd/system/elasticsearch_exporter-user-universe-main-dev.service
[2026-01-12 20:44:22 UTC] USER=www-data EUID=0 PID=709082 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-12 20:44:23 UTC] USER=www-data EUID=0 PID=709137 ACTION=passthru ARGS=systemctl enable elasticsearch_exporter-user-universe-main-dev.service
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  IP Conflict Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-universe-main-dev
IP Address:  10.100.1.219
Port:        9114
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔍 Checking IP conflict for user-universe-main-dev on 10.100.1.219:9114...
✅ IP 10.100.1.219:9114 is available - no conflicts detected

🔍 Checking for orphaned processes that might conflict...
✅ No orphaned processes detected

✅ All checks passed - safe to proceed with user-universe-main-dev setup
[2026-01-12 20:44:23 UTC] USER=www-data EUID=0 PID=709286 ACTION=passthru ARGS=systemctl restart elasticsearch_exporter-user-universe-main-dev.service
[OK]   elasticsearch_exporter configured on 10.100.1.219:9114
[INFO] Register this endpoint in metrics-user-universe-main-dev.fastorder.com scrape config
[OK]   ✓ elasticsearch_exporter integration complete

[INFO] 3.5️⃣ Configuring Prometheus to scrape Elasticsearch metrics...
[2026-01-12 20:44:26 UTC] USER=www-data EUID=0 PID=709404 ACTION=passthru ARGS=grep -q job_name: 'elasticsearch' /etc/prometheus/obs-user-universe-main-dev/prometheus.yml
[INFO] Adding Elasticsearch scrape target to Prometheus configuration...
[2026-01-12 20:44:26 UTC] USER=www-data EUID=0 PID=709427 ACTION=fsop ARGS=cp /etc/prometheus/obs-user-universe-main-dev/prometheus.yml /etc/prometheus/obs-user-universe-main-dev/prometheus.yml.backup-1768250666
[INFO] Created backup: /etc/prometheus/obs-user-universe-main-dev/prometheus.yml.backup-1768250666
[2026-01-12 20:44:26 UTC] USER=www-data EUID=0 PID=709475 ACTION=passthru ARGS=grep -q job_name: 'elasticsearch' /etc/prometheus/obs-user-universe-main-dev/prometheus.yml
[INFO] ✓ Elasticsearch job successfully inserted into config
[INFO] Validating Prometheus configuration with promtool...
Checking /etc/prometheus/obs-user-universe-main-dev/prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: /etc/prometheus/obs-user-universe-main-dev/prometheus.yml is valid prometheus config file syntax

Checking /etc/prometheus/obs-user-universe-main-dev/rules/basic_alerts.yml
  SUCCESS: 4 rules found

[OK]   ✓ Prometheus configuration validation PASSED
[OK]   ✓ Prometheus configuration updated successfully
[2026-01-12 20:44:27 UTC] USER=www-data EUID=0 PID=709544 ACTION=passthru ARGS=systemctl is-active --quiet prometheus-obs-user-universe-main-dev.service
[INFO] Reloading Prometheus configuration...
[2026-01-12 20:44:27 UTC] USER=www-data EUID=0 PID=709567 ACTION=passthru ARGS=systemctl restart prometheus-obs-user-universe-main-dev.service
[2026-01-12 20:44:30 UTC] USER=www-data EUID=0 PID=709658 ACTION=passthru ARGS=systemctl is-active --quiet prometheus-obs-user-universe-main-dev.service
[OK]   ✓ Prometheus reloaded successfully
[2026-01-12 20:44:30 UTC] USER=www-data EUID=0 PID=709679 ACTION=fsop ARGS=rm -f /tmp/prometheus_es_add.yml

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Elasticsearch Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Service: elasticsearch_exporter-user-universe-main-dev.service
[INFO] Metrics: http://localhost:9114/metrics
[INFO] Prometheus: https://metrics-user-universe-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-user-universe-main-dev.fastorder.com
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 4️⃣ Registering Elasticsearch nodes to monitoring database...
[INFO]    Constructed FQDN: search-user-universe-main-dev-elasticsearch-node-01.fastorder.com
[INFO] Registering: user-universe-main-dev-node-01
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Elasticsearch
[INFO]   Identifier:        user-universe-main-dev-node-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.219
[INFO]   Port:              9200
[INFO]   FQDN:              search-user-universe-main-dev-elasticsearch-node-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-universe-main-dev (service=user, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 7d81d573-27ab-4e5e-b4a8-cd63965b5ce5
[SUCCESS] Environment UUID: b683e8be-3b47-4d33-9d5c-389c63a0ae74
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b683e8be-3b47-4d33-9d5c-389c63a0ae74
[OK]   ✓ Registered: user-universe-main-dev-node-01
[OK]   ✓ Elasticsearch node registration completed successfully

[INFO] 5️⃣ Verifying monitoring integration...

[INFO] Checking elasticsearch_exporter service...
[OK]   ✓ elasticsearch_exporter-user-universe-main-dev.service is ACTIVE
[INFO] Checking Prometheus service...
[OK]   ✓ prometheus-obs-user-universe-main-dev.service is ACTIVE
[INFO] Validating Prometheus configuration...
[OK]   ✓ Prometheus configuration is VALID
[INFO] Checking Prometheus targets (waiting 35s for first scrape cycle)...
[2026-01-12 20:45:06 UTC] USER=www-data EUID=0 PID=712998 ACTION=passthru ARGS=grep -q tls_server_config /etc/prometheus/obs-user-universe-main-dev/web-config.yml
[OK]   ✓ Prometheus has Elasticsearch target configured
[OK]   ✓ Elasticsearch target is UP and being scraped

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ All monitoring integration steps completed
[INFO] ✅ All verifications PASSED
[INFO] ✅ Elasticsearch registered to dashboard database
[INFO] ✅ Prometheus scraping Elasticsearch metrics
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


✓ Step 4 completed successfully!

════════════════════════════════════════════════════════════════
🎉 All deployment tasks completed successfully!

✓ ✅ Search infrastructure (elasticsearch) setup completed successfully

[INFO] Loaded from topology.json: user-universe-main-dev
[2026-01-12 20:45:07] Loaded environment: user-universe-main-dev
[2026-01-12 20:45:07] Service: user, Zone: universe, Branch: main, Env: dev
[2026-01-12 20:45:07] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-12 20:45:07] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-12 20:45:07] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
[INFO] Starting finalizing setup process...
[INFO] Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps
[INFO] Environment: user-universe-main-dev

[INFO] Found 3 step(s) to execute

[INFO] 📦 Step 1/3: enable_disable_all_applications...
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
════════════════════════════════════════════════════════════════════════════════
  Environment Services Management
════════════════════════════════════════════════════════════════════════════════
  Environment:  user-universe-main-dev
  Action:       enable
  Triggered by: false
════════════════════════════════════════════════════════════════════════════════

🔍 Scanning for environment-specific services...
✅ Found 1 services for environment: user-universe-main-dev

📋 Services to enable:
────────────────────────────────────────────────────────────────────────────────
  • elasticsearch@user-universe-main-dev-node-01.service         [active/unmasked/enabled]
────────────────────────────────────────────────────────────────────────────────


❌ Cancelled by user
[OK] ✅ Step 1 completed: 01-enable_disable_all_applications.sh

[INFO] 📦 Step 2/3: verify monitoring...
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Monitoring Verification for user-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Detecting installed services...
Failed to print table: Broken pipe
[OK] ✓ Elasticsearch detected

[INFO] Services to verify: elasticsearch

[INFO] 2️⃣ Verifying exporters are running...
[OK] ✓ Elasticsearch exporter is running

[INFO] 3️⃣ Verifying Prometheus configuration...
[2026-01-12 20:45:10 UTC] USER=www-data EUID=0 PID=714396 ACTION=passthru ARGS=grep -q job_name: 'elasticsearch' /etc/prometheus/obs-user-universe-main-dev/prometheus.yml
[OK] ✓ elasticsearch is configured in Prometheus

[INFO] 4️⃣ Verifying Prometheus is actively scraping...
[OK] ✓ Prometheus is running
[OK] ✓ elasticsearch target is UP in Prometheus

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Monitoring Verification Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] All services are properly monitored!

[INFO] Monitoring Dashboard: https://skeleton.dev.fastorder.com/dashboard/monitoring

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Step 2 completed: 02-verify-monitoring.sh

[INFO] 📦 Step 3/3: register backup infrastructure...
[INFO] Loaded environment: user-universe-main-dev (svc=user zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔧 Registering Core Services & Backup Infrastructure for user-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Registering Main App...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Main App
[INFO]   Identifier:        user-universe-main-dev-main-app
[INFO]   Identifier Parent: application
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8080
[INFO]   FQDN:              app-user-universe-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-universe-main-dev (service=user, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 2d8cb171-8a2a-40f2-8723-a470b82d3039
[SUCCESS] Environment UUID: b683e8be-3b47-4d33-9d5c-389c63a0ae74
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b683e8be-3b47-4d33-9d5c-389c63a0ae74
/opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps/03-register-backup-infrastructure.sh: line 70: ok: command not found

[INFO] 2️⃣ Registering Audit Service...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Audit Service
[INFO]   Identifier:        user-universe-main-dev-audit
[INFO]   Identifier Parent: application
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8081
[INFO]   FQDN:              audit-user-universe-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       user-universe-main-dev (service=user, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 45744592-40ff-4948-9e1a-853652cc2165
[SUCCESS] Environment UUID: b683e8be-3b47-4d33-9d5c-389c63a0ae74
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b683e8be-3b47-4d33-9d5c-389c63a0ae74
/opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps/03-register-backup-infrastructure.sh: line 85: ok: command not found

[INFO] 3️⃣ Registering PostgreSQL Backup Node...
[ERROR] Invalid identifier format: backup-db
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register PostgreSQL backup node (non-blocking)

[INFO] 4️⃣ Registering Elasticsearch Backup Node...
[ERROR] Invalid identifier format: backup-search
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Elasticsearch backup node (non-blocking)

[INFO] 5️⃣ Registering Kafka Backup Node...
[ERROR] Invalid identifier format: backup-eventbus
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Kafka backup node (non-blocking)

[INFO] 6️⃣ Registering Backup Orchestrator...
[ERROR] Invalid identifier format: backup-orchestrator
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Backup orchestrator (non-blocking)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Core Services & Backup Infrastructure Registration Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Registered core services:
[INFO]   🚀 main-app            → Core application service
[INFO]   📋 audit               → Centralized audit logging (WORM)

[INFO] Registered backup nodes:
[INFO]   📦 backup-db           → PostgreSQL backup (pgBackRest, PITR)
[INFO]   📦 backup-search       → Elasticsearch snapshots (ILM, S3)
[INFO]   📦 backup-eventbus     → Kafka log segments (replication)
[INFO]   📦 backup-orchestrator → Central backup coordination

[INFO] Dashboard: https://skeleton.dev.fastorder.com/dashboard/monitoring
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Step 3 completed: 03-register-backup-infrastructure.sh


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ finalizing setup completed successfully!
[OK] Executed all 3 steps
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Environment: user-universe-main-dev
[INFO] Service: user
[INFO] Zone: universe
[INFO] Branch: main
[INFO] Env: dev