Environment: Web Universe Main Dev on web-03
"{\"env\": \"dev\", \"zone\": \"universe\", \"branch\": \"main\", \"db_app\": \"postgresql\", \"service\": \"web\", \"db_enabled\": true, \"pg_standby\": 0, \"pg_workers\": 1, \"description\": \"\", \"iam_enabled\": false, \"worker_1_ip\": \"\", \"eventbus_app\": \"kafka\", \"worker_1_fqdn\": \"db-web-universe-main-dev-postgresql-worker-01.fastorder.com\", \"search_enabled\": false, \"eventbus_enabled\": true, \"postgresql_enabled\": true, \"postgresql_run_verification\": true}"
This job encountered an error. You can restart from the failed step.
This job has been restarted. You are viewing an older attempt. The logs and status shown below are from the latest retry.
This job failed at one of the steps below. You can resume from where it failed to save time and avoid re-running successful steps.
[1m════════════════════════════════════════════════════════════════[0m
[1m FastOrder Pre-Flight Validation Checks[0m
[1m════════════════════════════════════════════════════════════════[0m
[0;34m[INFO][0m Checking SSH connectivity to target host...
[0;32m[✓][0m Target is localhost, skipping SSH check
[0;34m[INFO][0m Checking available disk space...
[0;34m[INFO][0m Checking /data disk (mounted separately for data storage)
[0;32m[✓][0m Disk space sufficient: 293GB available (required: 50GB)
[0;34m[INFO][0m Checking available memory...
[1;33m[⚠][0m Memory limited: 15GB (recommended: 16GB)
→ Consider reducing Elasticsearch nodes or PostgreSQL workers
[0;34m[INFO][0m Checking critical port availability...
[0;32m[✓][0m Port 5432 in use on specific IP (10.100.1.189:5432) - OK, can use different IP
[0;32m[✓][0m Port 9200 in use on specific IP ([::ffff:10.100.1.179]) - OK, can use different IP
[0;32m[✓][0m Port 9300 in use on specific IP ([::ffff:10.100.1.186]) - OK, can use different IP
[0;32m[✓][0m Port 9092 in use on specific IP ([::ffff:10.100.1.213]) - OK, can use different IP
[0;32m[✓][0m Port 2181 available (Zookeeper)
[0;34m[INFO][0m Checking DNS resolution...
[0;32m[✓][0m DNS resolution working: google.com
[0;32m[✓][0m DNS resolution working: github.com
[0;32m[✓][0m DNS resolution working: archive.ubuntu.com
[0;34m[INFO][0m Checking required system commands...
[0;32m[✓][0m Command available: curl
[0;32m[✓][0m Command available: wget
[0;32m[✓][0m Command available: git
[0;32m[✓][0m Command available: sudo
[0;32m[✓][0m Command available: systemctl
[0;32m[✓][0m Command available: apt-get
[0;34m[INFO][0m Checking current system load...
[1;33m[⚠][0m System load elevated: 4.20 (4 CPUs)
→ Provisioning may be slower than expected
[0;34m[INFO][0m Checking for existing environment conflicts...
[0;32m[✓][0m No conflicting services found for: web-uae-main-dev
[1m════════════════════════════════════════════════════════════════[0m
[1m Pre-Flight Check Summary[0m
[1m════════════════════════════════════════════════════════════════[0m
[1;33m[⚠][0m 2 warning(s) detected
⚠️ Environment can proceed with caution
Review warnings above and consider remediation
[INFO] Using web-provided environment: web-universe-main-dev
[INFO] Auto-creating state directory for web-universe-main-dev...
[ OK ] Created topology.json for web-universe-main-dev
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=10.100.1.51)
[0;36m[2026-02-05_03:40:13][0m Starting Terraform provisioning step
[0;36m[2026-02-05_03:40:13][0m Service: web
[0;36m[2026-02-05_03:40:13][0m Zone: universe
[0;36m[2026-02-05_03:40:13][0m Environment: dev
[0;36m[2026-02-05_03:40:13][0m Resource: web-03
[0;36m[2026-02-05_03:40:13][0m Terraform binary: /home/ab/bin/terraform
[0;36m[2026-02-05_03:40:13][0m HOME: /home/www-data
[0;36m[2026-02-05_03:40:13][0m AWS Config: /home/ab/.aws/config
[0;36m[2026-02-05_03:40:13][0m AWS Credentials: /home/ab/.aws/credentials
[0;36m[2026-02-05_03:40:13][0m Terraform directory: /opt/fastorder/cli/terraform/examples/citus-production
[0;36m[2026-02-05_03:40:13][0m Running terraform init...
[0m[1mInitializing the backend...[0m
[0m[1mUpgrading modules...[0m
- citus_cluster in ../../modules/citus_cluster
[0m[1mInitializing provider plugins...[0m
- Finding hashicorp/aws versions matching "~> 5.0"...
- Using previously-installed hashicorp/aws v5.100.0
[0m[1m[32mTerraform has been successfully initialized![0m[32m[0m
[0m[32m
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.[0m
[0;32m[2026-02-05_03:40:19] ✓[0m Terraform init succeeded
[0;36m[2026-02-05_03:40:19][0m Running terraform validate...
[32m[1mSuccess![0m The configuration is valid.
[0m
[0;32m[2026-02-05_03:40:22] ✓[0m Terraform validate succeeded
[0;36m[2026-02-05_03:40:22][0m Running terraform plan...
[0m[1mmodule.citus_cluster.data.aws_caller_identity.current: Reading...[0m[0m
[0m[1mmodule.citus_cluster.data.aws_caller_identity.current: Read complete after 0s [id=464621692046][0m
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
[32m+[0m create[0m
Terraform will perform the following actions:
[1m # module.citus_cluster.aws_iam_instance_profile.citus[0m will be created
[0m [32m+[0m[0m resource "aws_iam_instance_profile" "citus" {
[32m+[0m[0m arn = (known after apply)
[32m+[0m[0m create_date = (known after apply)
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m name = (known after apply)
[32m+[0m[0m name_prefix = "citus-prod-"
[32m+[0m[0m path = "/"
[32m+[0m[0m role = (known after apply)
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-prod"
}
[32m+[0m[0m tags_all = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "ManagedBy" = "Terraform"
[32m+[0m[0m "Name" = "citus-prod"
[32m+[0m[0m "Owner" = "Platform Team"
[32m+[0m[0m "Project" = "FastOrder"
}
[32m+[0m[0m unique_id = (known after apply)
}
[1m # module.citus_cluster.aws_iam_role.citus[0m will be created
[0m [32m+[0m[0m resource "aws_iam_role" "citus" {
[32m+[0m[0m arn = (known after apply)
[32m+[0m[0m assume_role_policy = jsonencode(
{
[32m+[0m[0m Statement = [
[32m+[0m[0m {
[32m+[0m[0m Action = "sts:AssumeRole"
[32m+[0m[0m Effect = "Allow"
[32m+[0m[0m Principal = {
[32m+[0m[0m Service = "ec2.amazonaws.com"
}
},
]
[32m+[0m[0m Version = "2012-10-17"
}
)
[32m+[0m[0m create_date = (known after apply)
[32m+[0m[0m force_detach_policies = false
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m managed_policy_arns = (known after apply)
[32m+[0m[0m max_session_duration = 3600
[32m+[0m[0m name = (known after apply)
[32m+[0m[0m name_prefix = "citus-prod-"
[32m+[0m[0m path = "/"
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-prod"
}
[32m+[0m[0m tags_all = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "ManagedBy" = "Terraform"
[32m+[0m[0m "Name" = "citus-prod"
[32m+[0m[0m "Owner" = "Platform Team"
[32m+[0m[0m "Project" = "FastOrder"
}
[32m+[0m[0m unique_id = (known after apply)
}
[1m # module.citus_cluster.aws_iam_role_policy.secrets_manager[0][0m will be created
[0m [32m+[0m[0m resource "aws_iam_role_policy" "secrets_manager" {
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m name = (known after apply)
[32m+[0m[0m name_prefix = "secrets-access-"
[32m+[0m[0m policy = jsonencode(
{
[32m+[0m[0m Statement = [
[32m+[0m[0m {
[32m+[0m[0m Action = [
[32m+[0m[0m "secretsmanager:GetSecretValue",
[32m+[0m[0m "secretsmanager:DescribeSecret",
]
[32m+[0m[0m Effect = "Allow"
[32m+[0m[0m Resource = "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/db/web/ksa/main/dev/postgresqladmin/ksa/prod*"
},
]
[32m+[0m[0m Version = "2012-10-17"
}
)
[32m+[0m[0m role = (known after apply)
}
[1m # module.citus_cluster.aws_iam_role_policy_attachment.cloudwatch[0m will be created
[0m [32m+[0m[0m resource "aws_iam_role_policy_attachment" "cloudwatch" {
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
[32m+[0m[0m role = (known after apply)
}
[1m # module.citus_cluster.aws_iam_role_policy_attachment.ssm[0m will be created
[0m [32m+[0m[0m resource "aws_iam_role_policy_attachment" "ssm" {
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
[32m+[0m[0m role = (known after apply)
}
[1m # module.citus_cluster.aws_instance.coordinator[0m will be created
[0m [32m+[0m[0m resource "aws_instance" "coordinator" {
[32m+[0m[0m ami = "ami-0b2aae5f4283c0df2"
[32m+[0m[0m arn = (known after apply)
[32m+[0m[0m associate_public_ip_address = (known after apply)
[32m+[0m[0m availability_zone = (known after apply)
[32m+[0m[0m cpu_core_count = (known after apply)
[32m+[0m[0m cpu_threads_per_core = (known after apply)
[32m+[0m[0m disable_api_stop = (known after apply)
[32m+[0m[0m disable_api_termination = (known after apply)
[32m+[0m[0m ebs_optimized = (known after apply)
[32m+[0m[0m enable_primary_ipv6 = (known after apply)
[32m+[0m[0m get_password_data = false
[32m+[0m[0m host_id = (known after apply)
[32m+[0m[0m host_resource_group_arn = (known after apply)
[32m+[0m[0m iam_instance_profile = (known after apply)
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m instance_initiated_shutdown_behavior = (known after apply)
[32m+[0m[0m instance_lifecycle = (known after apply)
[32m+[0m[0m instance_state = (known after apply)
[32m+[0m[0m instance_type = "r6i.2xlarge"
[32m+[0m[0m ipv6_address_count = (known after apply)
[32m+[0m[0m ipv6_addresses = (known after apply)
[32m+[0m[0m key_name = (known after apply)
[32m+[0m[0m monitoring = (known after apply)
[32m+[0m[0m outpost_arn = (known after apply)
[32m+[0m[0m password_data = (known after apply)
[32m+[0m[0m placement_group = (known after apply)
[32m+[0m[0m placement_partition_number = (known after apply)
[32m+[0m[0m primary_network_interface_id = (known after apply)
[32m+[0m[0m private_dns = (known after apply)
[32m+[0m[0m private_ip = (known after apply)
[32m+[0m[0m public_dns = (known after apply)
[32m+[0m[0m public_ip = (known after apply)
[32m+[0m[0m secondary_private_ips = (known after apply)
[32m+[0m[0m security_groups = (known after apply)
[32m+[0m[0m source_dest_check = true
[32m+[0m[0m spot_instance_request_id = (known after apply)
[32m+[0m[0m subnet_id = "subnet-0a1f5a9a74ed030cf"
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-coordinator-prod"
[32m+[0m[0m "Role" = "coordinator"
[32m+[0m[0m "Service" = "citus"
}
[32m+[0m[0m tags_all = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "ManagedBy" = "Terraform"
[32m+[0m[0m "Name" = "citus-coordinator-prod"
[32m+[0m[0m "Owner" = "Platform Team"
[32m+[0m[0m "Project" = "FastOrder"
[32m+[0m[0m "Role" = "coordinator"
[32m+[0m[0m "Service" = "citus"
}
[32m+[0m[0m tenancy = (known after apply)
[32m+[0m[0m user_data = "2a9e41ea765dcf3b3046ee10d2f458c18f00e430"
[32m+[0m[0m user_data_base64 = (known after apply)
[32m+[0m[0m user_data_replace_on_change = false
[32m+[0m[0m vpc_security_group_ids = (known after apply)
[32m+[0m[0m ebs_block_device {
[32m+[0m[0m delete_on_termination = false
[32m+[0m[0m device_name = "/dev/sdf"
[32m+[0m[0m encrypted = true
[32m+[0m[0m iops = 3000
[32m+[0m[0m kms_key_id = (known after apply)
[32m+[0m[0m snapshot_id = (known after apply)
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-coordinator-prod-data"
}
[32m+[0m[0m tags_all = (known after apply)
[32m+[0m[0m throughput = 125
[32m+[0m[0m volume_id = (known after apply)
[32m+[0m[0m volume_size = 500
[32m+[0m[0m volume_type = "gp3"
}
[32m+[0m[0m root_block_device {
[32m+[0m[0m delete_on_termination = false
[32m+[0m[0m device_name = (known after apply)
[32m+[0m[0m encrypted = true
[32m+[0m[0m iops = (known after apply)
[32m+[0m[0m kms_key_id = (known after apply)
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-coordinator-prod-root"
}
[32m+[0m[0m tags_all = (known after apply)
[32m+[0m[0m throughput = (known after apply)
[32m+[0m[0m volume_id = (known after apply)
[32m+[0m[0m volume_size = 100
[32m+[0m[0m volume_type = "gp3"
}
}
[1m # module.citus_cluster.aws_instance.workers[0][0m will be created
[0m [32m+[0m[0m resource "aws_instance" "workers" {
[32m+[0m[0m ami = "ami-0b2aae5f4283c0df2"
[32m+[0m[0m arn = (known after apply)
[32m+[0m[0m associate_public_ip_address = (known after apply)
[32m+[0m[0m availability_zone = (known after apply)
[32m+[0m[0m cpu_core_count = (known after apply)
[32m+[0m[0m cpu_threads_per_core = (known after apply)
[32m+[0m[0m disable_api_stop = (known after apply)
[32m+[0m[0m disable_api_termination = (known after apply)
[32m+[0m[0m ebs_optimized = (known after apply)
[32m+[0m[0m enable_primary_ipv6 = (known after apply)
[32m+[0m[0m get_password_data = false
[32m+[0m[0m host_id = (known after apply)
[32m+[0m[0m host_resource_group_arn = (known after apply)
[32m+[0m[0m iam_instance_profile = (known after apply)
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m instance_initiated_shutdown_behavior = (known after apply)
[32m+[0m[0m instance_lifecycle = (known after apply)
[32m+[0m[0m instance_state = (known after apply)
[32m+[0m[0m instance_type = "r6i.2xlarge"
[32m+[0m[0m ipv6_address_count = (known after apply)
[32m+[0m[0m ipv6_addresses = (known after apply)
[32m+[0m[0m key_name = (known after apply)
[32m+[0m[0m monitoring = (known after apply)
[32m+[0m[0m outpost_arn = (known after apply)
[32m+[0m[0m password_data = (known after apply)
[32m+[0m[0m placement_group = (known after apply)
[32m+[0m[0m placement_partition_number = (known after apply)
[32m+[0m[0m primary_network_interface_id = (known after apply)
[32m+[0m[0m private_dns = (known after apply)
[32m+[0m[0m private_ip = (known after apply)
[32m+[0m[0m public_dns = (known after apply)
[32m+[0m[0m public_ip = (known after apply)
[32m+[0m[0m secondary_private_ips = (known after apply)
[32m+[0m[0m security_groups = (known after apply)
[32m+[0m[0m source_dest_check = true
[32m+[0m[0m spot_instance_request_id = (known after apply)
[32m+[0m[0m subnet_id = "subnet-0a1f5a9a74ed030cf"
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-worker-0-prod"
[32m+[0m[0m "Role" = "worker"
[32m+[0m[0m "Service" = "citus"
[32m+[0m[0m "WorkerIndex" = "0"
}
[32m+[0m[0m tags_all = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "ManagedBy" = "Terraform"
[32m+[0m[0m "Name" = "citus-worker-0-prod"
[32m+[0m[0m "Owner" = "Platform Team"
[32m+[0m[0m "Project" = "FastOrder"
[32m+[0m[0m "Role" = "worker"
[32m+[0m[0m "Service" = "citus"
[32m+[0m[0m "WorkerIndex" = "0"
}
[32m+[0m[0m tenancy = (known after apply)
[32m+[0m[0m user_data = "7b4bd87c9982aab7fa463c8d12e99399661f8bde"
[32m+[0m[0m user_data_base64 = (known after apply)
[32m+[0m[0m user_data_replace_on_change = false
[32m+[0m[0m vpc_security_group_ids = (known after apply)
[32m+[0m[0m ebs_block_device {
[32m+[0m[0m delete_on_termination = false
[32m+[0m[0m device_name = "/dev/sdf"
[32m+[0m[0m encrypted = true
[32m+[0m[0m iops = 3000
[32m+[0m[0m kms_key_id = (known after apply)
[32m+[0m[0m snapshot_id = (known after apply)
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-worker-0-prod-data"
}
[32m+[0m[0m tags_all = (known after apply)
[32m+[0m[0m throughput = 125
[32m+[0m[0m volume_id = (known after apply)
[32m+[0m[0m volume_size = 500
[32m+[0m[0m volume_type = "gp3"
}
[32m+[0m[0m root_block_device {
[32m+[0m[0m delete_on_termination = false
[32m+[0m[0m device_name = (known after apply)
[32m+[0m[0m encrypted = true
[32m+[0m[0m iops = (known after apply)
[32m+[0m[0m kms_key_id = (known after apply)
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-worker-0-prod-root"
}
[32m+[0m[0m tags_all = (known after apply)
[32m+[0m[0m throughput = (known after apply)
[32m+[0m[0m volume_id = (known after apply)
[32m+[0m[0m volume_size = 100
[32m+[0m[0m volume_type = "gp3"
}
}
[1m # module.citus_cluster.aws_instance.workers[1][0m will be created
[0m [32m+[0m[0m resource "aws_instance" "workers" {
[32m+[0m[0m ami = "ami-0b2aae5f4283c0df2"
[32m+[0m[0m arn = (known after apply)
[32m+[0m[0m associate_public_ip_address = (known after apply)
[32m+[0m[0m availability_zone = (known after apply)
[32m+[0m[0m cpu_core_count = (known after apply)
[32m+[0m[0m cpu_threads_per_core = (known after apply)
[32m+[0m[0m disable_api_stop = (known after apply)
[32m+[0m[0m disable_api_termination = (known after apply)
[32m+[0m[0m ebs_optimized = (known after apply)
[32m+[0m[0m enable_primary_ipv6 = (known after apply)
[32m+[0m[0m get_password_data = false
[32m+[0m[0m host_id = (known after apply)
[32m+[0m[0m host_resource_group_arn = (known after apply)
[32m+[0m[0m iam_instance_profile = (known after apply)
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m instance_initiated_shutdown_behavior = (known after apply)
[32m+[0m[0m instance_lifecycle = (known after apply)
[32m+[0m[0m instance_state = (known after apply)
[32m+[0m[0m instance_type = "r6i.2xlarge"
[32m+[0m[0m ipv6_address_count = (known after apply)
[32m+[0m[0m ipv6_addresses = (known after apply)
[32m+[0m[0m key_name = (known after apply)
[32m+[0m[0m monitoring = (known after apply)
[32m+[0m[0m outpost_arn = (known after apply)
[32m+[0m[0m password_data = (known after apply)
[32m+[0m[0m placement_group = (known after apply)
[32m+[0m[0m placement_partition_number = (known after apply)
[32m+[0m[0m primary_network_interface_id = (known after apply)
[32m+[0m[0m private_dns = (known after apply)
[32m+[0m[0m private_ip = (known after apply)
[32m+[0m[0m public_dns = (known after apply)
[32m+[0m[0m public_ip = (known after apply)
[32m+[0m[0m secondary_private_ips = (known after apply)
[32m+[0m[0m security_groups = (known after apply)
[32m+[0m[0m source_dest_check = true
[32m+[0m[0m spot_instance_request_id = (known after apply)
[32m+[0m[0m subnet_id = "subnet-02c930351cde1e9c3"
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-worker-1-prod"
[32m+[0m[0m "Role" = "worker"
[32m+[0m[0m "Service" = "citus"
[32m+[0m[0m "WorkerIndex" = "1"
}
[32m+[0m[0m tags_all = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "ManagedBy" = "Terraform"
[32m+[0m[0m "Name" = "citus-worker-1-prod"
[32m+[0m[0m "Owner" = "Platform Team"
[32m+[0m[0m "Project" = "FastOrder"
[32m+[0m[0m "Role" = "worker"
[32m+[0m[0m "Service" = "citus"
[32m+[0m[0m "WorkerIndex" = "1"
}
[32m+[0m[0m tenancy = (known after apply)
[32m+[0m[0m user_data = "7b4bd87c9982aab7fa463c8d12e99399661f8bde"
[32m+[0m[0m user_data_base64 = (known after apply)
[32m+[0m[0m user_data_replace_on_change = false
[32m+[0m[0m vpc_security_group_ids = (known after apply)
[32m+[0m[0m ebs_block_device {
[32m+[0m[0m delete_on_termination = false
[32m+[0m[0m device_name = "/dev/sdf"
[32m+[0m[0m encrypted = true
[32m+[0m[0m iops = 3000
[32m+[0m[0m kms_key_id = (known after apply)
[32m+[0m[0m snapshot_id = (known after apply)
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-worker-1-prod-data"
}
[32m+[0m[0m tags_all = (known after apply)
[32m+[0m[0m throughput = 125
[32m+[0m[0m volume_id = (known after apply)
[32m+[0m[0m volume_size = 500
[32m+[0m[0m volume_type = "gp3"
}
[32m+[0m[0m root_block_device {
[32m+[0m[0m delete_on_termination = false
[32m+[0m[0m device_name = (known after apply)
[32m+[0m[0m encrypted = true
[32m+[0m[0m iops = (known after apply)
[32m+[0m[0m kms_key_id = (known after apply)
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-worker-1-prod-root"
}
[32m+[0m[0m tags_all = (known after apply)
[32m+[0m[0m throughput = (known after apply)
[32m+[0m[0m volume_id = (known after apply)
[32m+[0m[0m volume_size = 100
[32m+[0m[0m volume_type = "gp3"
}
}
[1m # module.citus_cluster.aws_security_group.citus[0m will be created
[0m [32m+[0m[0m resource "aws_security_group" "citus" {
[32m+[0m[0m arn = (known after apply)
[32m+[0m[0m description = "Security group for Citus cluster"
[32m+[0m[0m egress = [
[32m+[0m[0m {
[32m+[0m[0m cidr_blocks = [
[32m+[0m[0m "0.0.0.0/0",
]
[32m+[0m[0m description = "Allow all outbound"
[32m+[0m[0m from_port = 0
[32m+[0m[0m ipv6_cidr_blocks = []
[32m+[0m[0m prefix_list_ids = []
[32m+[0m[0m protocol = "-1"
[32m+[0m[0m security_groups = []
[32m+[0m[0m self = false
[32m+[0m[0m to_port = 0
},
]
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m ingress = [
[32m+[0m[0m {
[32m+[0m[0m cidr_blocks = [
[32m+[0m[0m "10.0.0.0/8",
]
[32m+[0m[0m description = "PgBouncer access"
[32m+[0m[0m from_port = 6432
[32m+[0m[0m ipv6_cidr_blocks = []
[32m+[0m[0m prefix_list_ids = []
[32m+[0m[0m protocol = "tcp"
[32m+[0m[0m security_groups = []
[32m+[0m[0m self = false
[32m+[0m[0m to_port = 6432
},
[32m+[0m[0m {
[32m+[0m[0m cidr_blocks = [
[32m+[0m[0m "10.0.0.0/8",
]
[32m+[0m[0m description = "PostgreSQL access"
[32m+[0m[0m from_port = 5432
[32m+[0m[0m ipv6_cidr_blocks = []
[32m+[0m[0m prefix_list_ids = []
[32m+[0m[0m protocol = "tcp"
[32m+[0m[0m security_groups = []
[32m+[0m[0m self = false
[32m+[0m[0m to_port = 5432
},
[32m+[0m[0m {
[32m+[0m[0m cidr_blocks = [
[32m+[0m[0m "10.0.0.0/8",
]
[32m+[0m[0m description = "SSH access"
[32m+[0m[0m from_port = 22
[32m+[0m[0m ipv6_cidr_blocks = []
[32m+[0m[0m prefix_list_ids = []
[32m+[0m[0m protocol = "tcp"
[32m+[0m[0m security_groups = []
[32m+[0m[0m self = false
[32m+[0m[0m to_port = 22
},
[32m+[0m[0m {
[32m+[0m[0m cidr_blocks = []
[32m+[0m[0m description = "Internal cluster communication"
[32m+[0m[0m from_port = 0
[32m+[0m[0m ipv6_cidr_blocks = []
[32m+[0m[0m prefix_list_ids = []
[32m+[0m[0m protocol = "tcp"
[32m+[0m[0m security_groups = []
[32m+[0m[0m self = true
[32m+[0m[0m to_port = 65535
},
]
[32m+[0m[0m name = (known after apply)
[32m+[0m[0m name_prefix = "citus-prod-"
[32m+[0m[0m owner_id = (known after apply)
[32m+[0m[0m revoke_rules_on_delete = false
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-prod"
[32m+[0m[0m "Service" = "citus"
}
[32m+[0m[0m tags_all = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "ManagedBy" = "Terraform"
[32m+[0m[0m "Name" = "citus-prod"
[32m+[0m[0m "Owner" = "Platform Team"
[32m+[0m[0m "Project" = "FastOrder"
[32m+[0m[0m "Service" = "citus"
}
[32m+[0m[0m vpc_id = "vpc-0af7da1e7d94d62bd"
}
[1mPlan:[0m 9 to add, 0 to change, 0 to destroy.
[0m
Changes to Outputs:
[32m+[0m[0m connection_string = (sensitive value)
[32m+[0m[0m coordinator_ip = (known after apply)
[32m+[0m[0m worker_ips = [
[32m+[0m[0m (known after apply),
[32m+[0m[0m (known after apply),
]
[90m
─────────────────────────────────────────────────────────────────────────────[0m
Saved the plan to: tfplan
To perform exactly these actions, run the following command to apply:
terraform apply "tfplan"
[0;32m[2026-02-05_03:40:26] ✓[0m Terraform plan succeeded
[0;36m[2026-02-05_03:40:26][0m Generating plan JSON...
[0;32m[2026-02-05_03:40:29] ✓[0m Terraform provisioning step completed successfully
Next step: Review the plan and apply with 'terraform apply tfplan'
[INFO] FastOrder Environment Preparation
[INFO] Service: web
[INFO] Zone: universe
[INFO] Environment: dev
[INFO] Branch: main
[INFO] State Directory: /opt/fastorder/bash/scripts/env_app_setup/state
[INFO] Library: /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] IP: 142.93.238.16 (specified)
[INFO] Creating environment using fo-env...
[INFO] Creating new FastOrder environment (v1 topology)
[INFO] Generated environment ID: web-universe-main-dev
[INFO] Using provided IP: 142.93.238.16
[INFO] Allocated interface: eth0:16
[INFO] Configuring network interface for VM IP: 142.93.238.16
[INFO] VM IP 142.93.238.16 is already configured on eth0:16
[CONFIG] No web configuration found for environment: web-universe-main-dev
[CONFIG] Using defaults: ES_NODES=3, PG_WORKERS=1
[INFO] Service enabled flags: db=yes, eventbus=yes, search=no
[ OK ] Created topology.json at /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/topology.json
[ OK ] Generated overlay configurations in /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/generated/
[ OK ] Updated environments.json
[ OK ] Updated setup.json
[ OK ] Environment created successfully!
[INFO]
[INFO] Environment Details:
[INFO] ID: web-universe-main-dev
[INFO] Service: web
[INFO] zone: universe
[INFO] Environment: dev
[INFO] Branch: main
[INFO] IP: 142.93.238.16
[INFO] Interface: eth0:16
[INFO]
[INFO] Configuration files:
[INFO] Topology: /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/topology.json
[INFO] Generated: /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/generated/*.env
[INFO] Overrides: /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/overrides/*.env
[INFO]
[INFO] To use this environment:
[INFO] export ENV_ID="web-universe-main-dev"
[INFO] source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO] init_environment
[ OK ] Environment preparation completed successfully!
[INFO] Creating topology from web form submission...
[INFO] Using environment from web interface: web-universe-main-dev
[0;32m[2026-02-05 03:40:32][0m Using web-provided environment: web-universe-main-dev
[0;32m[2026-02-05 03:40:32][0m Service: web, Zone: universe, Branch: main, Env: dev
[ OK ] Environment initialized successfully (mode: general)
[INFO] Creating topology.json from web form submission...
[INFO] DEBUG: Service enabled flags...
[INFO] DB_ENABLED=yes
[INFO] EVENTBUS_ENABLED=yes
[INFO] SEARCH_ENABLED=no
[INFO] DEBUG: Checking for form submission variables...
[INFO] service_es_ip=NOT SET
[INFO] service_es_fqdn=NOT SET
[INFO] service_pg_coordinator_ip=NOT SET
[INFO] Skipping Elasticsearch/Search - SEARCH_ENABLED=no
[ OK ] Topology created from form data
[INFO] Applications registered:
[ OK ] Topology created from form data
[INFO] Next steps:
[INFO] 1. Review the generated topology.json and configurations
[INFO] 2. Customize overrides/*.env files if needed
[INFO] 3. Run subsequent installation steps (02-install-postgresql, etc.)
[INFO] To use this environment in other scripts:
[INFO] export ENV_ID="$(fo-env list | tail -n1 | awk '{print $1}')"
[INFO] source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO] init_environment
⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 🚀 OBSERVABILITY CELL PROVISIONING STARTED
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Script: 02-observability-cell/run.sh
[0;34m[INFO][0m Timestamp: 2026-02-05 03:40:32 UTC
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Ensuring correct permissions for observability deployment...
[2026-02-05 03:40:32 UTC] USER=www-data EUID=0 PID=3556003 ACTION=fsop ARGS=chmod 775 /var/log/fastorder
[2026-02-05 03:40:32 UTC] USER=www-data EUID=0 PID=3556014 ACTION=fsop ARGS=chown www-data:www-data /var/log/fastorder
[2026-02-05 03:40:32 UTC] USER=www-data EUID=0 PID=3556040 ACTION=fsop ARGS=touch /var/log/fastorder/provisioning-elevated.log
[2026-02-05 03:40:32 UTC] USER=www-data EUID=0 PID=3556051 ACTION=fsop ARGS=chmod 666 /var/log/fastorder/provisioning-elevated.log
[2026-02-05 03:40:32 UTC] USER=www-data EUID=0 PID=3556062 ACTION=fsop ARGS=chown www-data:www-data /var/log/fastorder/provisioning-elevated.log
[0;32m[OK][0m Log directory: /var/log/fastorder (775)
[0;32m[OK][0m Log file: provisioning-elevated.log (666)
[2026-02-05 03:40:32 UTC] USER=www-data EUID=0 PID=3556071 ACTION=fsop ARGS=chmod 775 /opt/fastorder/bash/scripts/env_app_setup/state
[0;32m[OK][0m State directory: 775
[2026-02-05 03:40:32 UTC] USER=www-data EUID=0 PID=3556080 ACTION=fsop ARGS=mkdir -p /etc/fastorder/observability/certs
[2026-02-05 03:40:32 UTC] USER=www-data EUID=0 PID=3556089 ACTION=fsop ARGS=chmod 750 /etc/fastorder/observability/certs
[0;32m[OK][0m Cert directory: /etc/fastorder/observability/certs (750 - secure)
[0;32m[OK][0m Lib scripts: executable (755)
[0;32m[OK][0m All deployment scripts: executable (755)
[0;32m[OK][0m All directories: accessible (755)
[0;32m[OK][0m ✅ All permissions verified and fixed
[0;34m[CREDS][0m Using AWS credentials from: /var/www/.aws/credentials
[0;34m[CREDS][0m Credential management library loaded (region: me-central-1)
[INFO] Using web-provided environment: web-universe-main-dev
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
═══════════════════════════════════════════════════════════════════════════════
OBSERVABILITY CELL PROVISIONING
═══════════════════════════════════════════════════════════════════════════════
[INFO] Application Cell: web-universe-main-dev
[INFO] Observability Cell: obs-web-universe-main-dev
[INFO] Service: web | Zone: universe | Env: dev
[INFO] Step 1/10: Provisioning network infrastructure...
[INFO] Allocated new IP for obs: 10.100.1.235
[2026-02-05 03:40:33 UTC] USER=www-data EUID=0 PID=3556635 ACTION=fsop ARGS=cp /tmp/tmp.uplutFE9eF /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/topology.json
[INFO] Allocated new IP for metrics: 10.100.1.236
[2026-02-05 03:40:34 UTC] USER=www-data EUID=0 PID=3556662 ACTION=fsop ARGS=cp /tmp/tmp.hqoIAmmSfz /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/topology.json
[INFO] Allocated new IP for dashboards: 10.100.1.237
[2026-02-05 03:40:34 UTC] USER=www-data EUID=0 PID=3556679 ACTION=fsop ARGS=cp /tmp/tmp.0rg8t7cVQv /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/topology.json
[INFO] Allocated new IP for logstore: 10.100.1.238
[2026-02-05 03:40:34 UTC] USER=www-data EUID=0 PID=3556696 ACTION=fsop ARGS=cp /tmp/tmp.rH0m2Plfcg /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/topology.json
[INFO] Allocated new IP for traces: 10.100.1.239
[2026-02-05 03:40:34 UTC] USER=www-data EUID=0 PID=3556713 ACTION=fsop ARGS=cp /tmp/tmp.U67boxqEW7 /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/topology.json
[INFO] Allocated new IP for alerts: 10.100.1.240
[2026-02-05 03:40:34 UTC] USER=www-data EUID=0 PID=3556730 ACTION=fsop ARGS=cp /tmp/tmp.LJ9LcKDh1a /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/topology.json
[INFO] Allocated new IP for telemetry: 10.100.1.241
[2026-02-05 03:40:34 UTC] USER=www-data EUID=0 PID=3556747 ACTION=fsop ARGS=cp /tmp/tmp.fU9WJ3iUpF /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/topology.json
[INFO] Allocated observability IPs:
[INFO] metrics: 10.100.1.236
[INFO] alerts: 10.100.1.240
[INFO] dashboards: 10.100.1.237
[INFO] traces: 10.100.1.239
[INFO] telemetry: 10.100.1.241
[INFO] logstore: 10.100.1.238
[INFO] proxy: 10.100.1.235
[INFO] obs: 10.100.1.235
[ OK ] Network infrastructure allocated
[INFO] Cleaning up ports from previous environments...
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking and cleaning ports for observability cell: obs-web-universe-main-dev
[0;34m[INFO][0m IP Address: 10.100.1.235
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking for conflicting observability services...
[0;34m[INFO][0m Service clickhouse-server-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service prometheus-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service grafana-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service tempo-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service alertmanager-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Found 11 observability service(s) (all belong to current cell)
[0;34m[INFO][0m Checking for remaining processes on IP 10.100.1.235...
[0;34m[INFO][0m Scanning 15 ports...
[0;32m[OK][0m ✅ All 15 ports are FREE - ready for installation
[0;32m[OK][0m Port cleanup completed successfully
[0;34m[INFO][0m Configuring IP aliases on network interface...
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING NETWORK IP ALIASES
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Interface: lo
[0;34m[INFO][0m IP Count: 8
[0;34m[INFO][0m Configuring: metrics → 10.100.1.236
[0;34m[INFO][0m Configuring IP alias: 10.100.1.236/32 on lo
[0;32m[OK][0m ✅ IP 10.100.1.236 configured successfully on lo
[0;32m[OK][0m ✅ IP 10.100.1.236 verified on network interface
[0;34m[INFO][0m Configuring: alerts → 10.100.1.240
[0;34m[INFO][0m Configuring IP alias: 10.100.1.240/32 on lo
[0;32m[OK][0m ✅ IP 10.100.1.240 configured successfully on lo
[0;32m[OK][0m ✅ IP 10.100.1.240 verified on network interface
[0;34m[INFO][0m Configuring: dashboards → 10.100.1.237
[0;34m[INFO][0m Configuring IP alias: 10.100.1.237/32 on lo
[0;32m[OK][0m ✅ IP 10.100.1.237 configured successfully on lo
[0;32m[OK][0m ✅ IP 10.100.1.237 verified on network interface
[0;34m[INFO][0m Configuring: traces → 10.100.1.239
[0;34m[INFO][0m Configuring IP alias: 10.100.1.239/32 on lo
[0;32m[OK][0m ✅ IP 10.100.1.239 configured successfully on lo
[0;32m[OK][0m ✅ IP 10.100.1.239 verified on network interface
[0;34m[INFO][0m Configuring: telemetry → 10.100.1.241
[0;34m[INFO][0m Configuring IP alias: 10.100.1.241/32 on lo
[0;32m[OK][0m ✅ IP 10.100.1.241 configured successfully on lo
[0;32m[OK][0m ✅ IP 10.100.1.241 verified on network interface
[0;34m[INFO][0m Configuring: logstore → 10.100.1.238
[0;34m[INFO][0m IP 10.100.1.238 already configured on network interface
[0;34m[INFO][0m Configuring: proxy → 10.100.1.235
[0;34m[INFO][0m Configuring IP alias: 10.100.1.235/32 on lo
[0;32m[OK][0m ✅ IP 10.100.1.235 configured successfully on lo
[0;32m[OK][0m ✅ IP 10.100.1.235 verified on network interface
[0;34m[INFO][0m Configuring: obs → 10.100.1.235
[0;34m[INFO][0m IP 10.100.1.235 already configured on network interface
[0;32m[OK][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ All IP aliases configured successfully
[0;32m[OK][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Current IP configuration on lo:
inet 127.0.0.1/8 scope host lo
inet 10.100.1.186/32 scope global lo:186
inet 10.100.1.192/32 scope global lo:pgbouncer
inet 10.100.1.184/32 scope global lo:pgbouncer
inet 10.100.1.204/32 scope global lo:pgbouncer
inet 10.100.60.2/32 scope global lo
inet 10.100.1.236/32 scope global lo
inet 10.100.1.240/32 scope global lo
inet 10.100.1.237/32 scope global lo
inet 10.100.1.239/32 scope global lo
inet 10.100.1.241/32 scope global lo
inet 10.100.1.235/32 scope global lo
[0;32m[OK][0m IP aliases configured on network interface
[0;34m[INFO][0m Step 2/10: Creating DNS entries...
[0;34m[INFO][0m Configuring DNS entries in /etc/hosts...
[0;34m[INFO][0m Added: metrics-web-universe-main-dev-prometheus.fastorder.com → 10.100.1.236
[0;34m[INFO][0m Added: alerts-web-universe-main-dev-alertmanager.fastorder.com → 10.100.1.240
[0;34m[INFO][0m Added: dashboards-web-universe-main-dev-grafana.fastorder.com → 10.100.1.237
[0;34m[INFO][0m Added: traces-web-universe-main-dev-tempo.fastorder.com → 10.100.1.239
[0;34m[INFO][0m Added: telemetry-web-universe-main-dev-opentelemetry.fastorder.com → 10.100.1.241
[0;34m[INFO][0m Added: logstore-web-universe-main-dev-clickhouse.fastorder.com → 10.100.1.238
[0;34m[INFO][0m Added: observe-web-universe-main-dev.fastorder.com → 10.100.1.235
[0;34m[INFO][0m Adding observability integration aliases...
[0;34m[INFO][0m Added alias: metrics-web-universe-main-dev.fastorder.com → 10.100.1.236
[0;34m[INFO][0m Added alias: alerts-web-universe-main-dev.fastorder.com → 10.100.1.240
[0;34m[INFO][0m Added alias: dashboards-web-universe-main-dev.fastorder.com → 10.100.1.237
[0;34m[INFO][0m Added alias: traces-web-universe-main-dev.fastorder.com → 10.100.1.239
[0;34m[INFO][0m Added alias: telemetry-web-universe-main-dev.fastorder.com → 10.100.1.241
[0;34m[INFO][0m Added alias: logstore-web-universe-main-dev.fastorder.com → 10.100.1.238
[2026-02-05 03:40:35 UTC] USER=www-data EUID=0 PID=3557169 ACTION=fsop ARGS=sed -i /observe-web-universe-main-dev.fastorder.com/d /etc/hosts
[0;34m[INFO][0m Added alias: observe-web-universe-main-dev.fastorder.com → 10.100.1.235
[0;32m[OK][0m DNS entries created
[0;34m[INFO][0m Step 3/10: Creating AWS Secrets Manager structure...
[INFO] Creating AWS Secrets Manager structure
[INFO] Base path: fastorder/observability/web/universe/dev
[INFO] Observability Cell: obs-web-universe-main-dev
[INFO] Application Cell: web-universe-main-dev
[INFO] Creating: fastorder/observability/web/universe/dev/metrics
[INFO] Creating: fastorder/observability/web/universe/dev/dashboards
[INFO] Creating: fastorder/observability/web/universe/dev/logstore
[INFO] Creating: fastorder/observability/web/universe/dev/traces
[INFO] Creating: fastorder/observability/web/universe/dev/telemetry
[INFO] Creating: fastorder/observability/web/universe/dev/alerts
[INFO] Secrets structure created successfully
[0;32m[OK][0m Secrets structure created
[0;34m[INFO][0m Step 4/10: Generating mTLS certificates...
[INFO] Generating mTLS certificates for observability cell
[INFO] Observability Cell: obs-web-universe-main-dev
[INFO] Components: prometheus,grafana,loki,tempo,otlp_collector,clickhouse,alertmanager
[INFO] Creating certificate directory: /etc/fastorder/observability/certs/obs-web-universe-main-dev
[2026-02-05 03:40:56 UTC] USER=www-data EUID=0 PID=3557796 ACTION=fsop ARGS=mkdir -p /etc/fastorder/observability/certs/obs-web-universe-main-dev
[2026-02-05 03:40:56 UTC] USER=www-data EUID=0 PID=3557805 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-web-universe-main-dev
[INFO] Generating CA certificate for obs-web-universe-main-dev
[2026-02-05 03:40:56 UTC] USER=www-data EUID=0 PID=3557814 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-key.pem 4096
[2026-02-05 03:40:58 UTC] USER=www-data EUID=0 PID=3557855 ACTION=fsop ARGS=openssl req -new -x509 -days 3650 -key /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-key.pem -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=obs-web-universe-main-dev-ca
[2026-02-05 03:40:58 UTC] USER=www-data EUID=0 PID=3557864 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-key.pem
[2026-02-05 03:40:58 UTC] USER=www-data EUID=0 PID=3557873 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem
[INFO] CA certificate created: /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem
[INFO] Generating certificate for: prometheus
[2026-02-05 03:40:58 UTC] USER=www-data EUID=0 PID=3557882 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/prometheus-key.pem 2048
[2026-02-05 03:40:58 UTC] USER=www-data EUID=0 PID=3557893 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-web-universe-main-dev/prometheus-key.pem -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/prometheus-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=prometheus.obs-web-universe-main-dev
[2026-02-05 03:40:58 UTC] USER=www-data EUID=0 PID=3557902 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-web-universe-main-dev/prometheus-csr.pem -CA /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/prometheus-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = prometheus.obs-web-universe-main-dev
[2026-02-05 03:40:58 UTC] USER=www-data EUID=0 PID=3557911 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-web-universe-main-dev/prometheus-key.pem
[2026-02-05 03:40:58 UTC] USER=www-data EUID=0 PID=3557920 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/prometheus-cert.pem
[2026-02-05 03:40:58 UTC] USER=www-data EUID=0 PID=3557929 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-web-universe-main-dev/prometheus-csr.pem
[INFO] Certificate created: /etc/fastorder/observability/certs/obs-web-universe-main-dev/prometheus-cert.pem
[INFO] Generating certificate for: grafana
[2026-02-05 03:40:59 UTC] USER=www-data EUID=0 PID=3557943 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/grafana-key.pem 2048
[2026-02-05 03:40:59 UTC] USER=www-data EUID=0 PID=3557954 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-web-universe-main-dev/grafana-key.pem -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/grafana-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=grafana.obs-web-universe-main-dev
[2026-02-05 03:40:59 UTC] USER=www-data EUID=0 PID=3557963 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-web-universe-main-dev/grafana-csr.pem -CA /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/grafana-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = grafana.obs-web-universe-main-dev
[2026-02-05 03:40:59 UTC] USER=www-data EUID=0 PID=3557972 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-web-universe-main-dev/grafana-key.pem
[2026-02-05 03:40:59 UTC] USER=www-data EUID=0 PID=3557981 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/grafana-cert.pem
[2026-02-05 03:40:59 UTC] USER=www-data EUID=0 PID=3557990 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-web-universe-main-dev/grafana-csr.pem
[INFO] Certificate created: /etc/fastorder/observability/certs/obs-web-universe-main-dev/grafana-cert.pem
[INFO] Generating certificate for: loki
[2026-02-05 03:40:59 UTC] USER=www-data EUID=0 PID=3557999 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/loki-key.pem 2048
[2026-02-05 03:40:59 UTC] USER=www-data EUID=0 PID=3558022 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-web-universe-main-dev/loki-key.pem -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/loki-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=loki.obs-web-universe-main-dev
[2026-02-05 03:40:59 UTC] USER=www-data EUID=0 PID=3558043 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-web-universe-main-dev/loki-csr.pem -CA /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/loki-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = loki.obs-web-universe-main-dev
[2026-02-05 03:40:59 UTC] USER=www-data EUID=0 PID=3558058 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-web-universe-main-dev/loki-key.pem
[2026-02-05 03:40:59 UTC] USER=www-data EUID=0 PID=3558067 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/loki-cert.pem
[2026-02-05 03:40:59 UTC] USER=www-data EUID=0 PID=3558076 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-web-universe-main-dev/loki-csr.pem
[INFO] Certificate created: /etc/fastorder/observability/certs/obs-web-universe-main-dev/loki-cert.pem
[INFO] Generating certificate for: tempo
[2026-02-05 03:40:59 UTC] USER=www-data EUID=0 PID=3558085 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/tempo-key.pem 2048
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558096 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-web-universe-main-dev/tempo-key.pem -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/tempo-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=tempo.obs-web-universe-main-dev
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558105 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-web-universe-main-dev/tempo-csr.pem -CA /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/tempo-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = tempo.obs-web-universe-main-dev
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558114 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-web-universe-main-dev/tempo-key.pem
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558123 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/tempo-cert.pem
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558132 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-web-universe-main-dev/tempo-csr.pem
[INFO] Certificate created: /etc/fastorder/observability/certs/obs-web-universe-main-dev/tempo-cert.pem
[INFO] Generating certificate for: otlp_collector
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558141 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/otlp_collector-key.pem 2048
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558150 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-web-universe-main-dev/otlp_collector-key.pem -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/otlp_collector-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=otlp_collector.obs-web-universe-main-dev
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558159 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-web-universe-main-dev/otlp_collector-csr.pem -CA /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/otlp_collector-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = otlp_collector.obs-web-universe-main-dev
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558168 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-web-universe-main-dev/otlp_collector-key.pem
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558177 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/otlp_collector-cert.pem
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558186 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-web-universe-main-dev/otlp_collector-csr.pem
[INFO] Certificate created: /etc/fastorder/observability/certs/obs-web-universe-main-dev/otlp_collector-cert.pem
[INFO] Generating certificate for: clickhouse
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558195 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/clickhouse-key.pem 2048
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558204 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-web-universe-main-dev/clickhouse-key.pem -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/clickhouse-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=clickhouse.obs-web-universe-main-dev
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558213 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-web-universe-main-dev/clickhouse-csr.pem -CA /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/clickhouse-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = clickhouse.obs-web-universe-main-dev
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558222 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-web-universe-main-dev/clickhouse-key.pem
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558231 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/clickhouse-cert.pem
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558240 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-web-universe-main-dev/clickhouse-csr.pem
[INFO] Certificate created: /etc/fastorder/observability/certs/obs-web-universe-main-dev/clickhouse-cert.pem
[INFO] Generating certificate for: alertmanager
[2026-02-05 03:41:00 UTC] USER=www-data EUID=0 PID=3558249 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/alertmanager-key.pem 2048
[2026-02-05 03:41:01 UTC] USER=www-data EUID=0 PID=3558288 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-web-universe-main-dev/alertmanager-key.pem -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/alertmanager-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=alertmanager.obs-web-universe-main-dev
[2026-02-05 03:41:01 UTC] USER=www-data EUID=0 PID=3558300 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-web-universe-main-dev/alertmanager-csr.pem -CA /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/alertmanager-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = alertmanager.obs-web-universe-main-dev
[2026-02-05 03:41:01 UTC] USER=www-data EUID=0 PID=3558325 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-web-universe-main-dev/alertmanager-key.pem
[2026-02-05 03:41:01 UTC] USER=www-data EUID=0 PID=3558334 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/alertmanager-cert.pem
[2026-02-05 03:41:01 UTC] USER=www-data EUID=0 PID=3558343 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-web-universe-main-dev/alertmanager-csr.pem
[INFO] Certificate created: /etc/fastorder/observability/certs/obs-web-universe-main-dev/alertmanager-cert.pem
[INFO] Generating PHP client certificate for metrics service...
[2026-02-05 03:41:01 UTC] USER=www-data EUID=0 PID=3558352 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/php-client-key.pem 2048
[2026-02-05 03:41:01 UTC] USER=www-data EUID=0 PID=3558389 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-web-universe-main-dev/php-client-key.pem -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/php-client-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Dashboard/CN=php-metrics-client.obs-web-universe-main-dev
[2026-02-05 03:41:01 UTC] USER=www-data EUID=0 PID=3558400 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-web-universe-main-dev/php-client-csr.pem -CA /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/php-client-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Dashboard, CN = php-metrics-client.obs-web-universe-main-dev
[2026-02-05 03:41:01 UTC] USER=www-data EUID=0 PID=3558409 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-web-universe-main-dev/php-client-key.pem
[2026-02-05 03:41:01 UTC] USER=www-data EUID=0 PID=3558418 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/php-client-cert.pem
[2026-02-05 03:41:01 UTC] USER=www-data EUID=0 PID=3558427 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-web-universe-main-dev/php-client-key.pem
[2026-02-05 03:41:01 UTC] USER=www-data EUID=0 PID=3558436 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-web-universe-main-dev/php-client-cert.pem
[2026-02-05 03:41:01 UTC] USER=www-data EUID=0 PID=3558445 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-web-universe-main-dev/php-client-csr.pem
[INFO] PHP client certificate created: /etc/fastorder/observability/certs/obs-web-universe-main-dev/php-client-cert.pem
[INFO] Generating Apache client certificate for mTLS reverse proxy...
[2026-02-05 03:41:01 UTC] USER=www-data EUID=0 PID=3558454 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/apache-client-key.pem 2048
[2026-02-05 03:41:02 UTC] USER=www-data EUID=0 PID=3558465 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-web-universe-main-dev/apache-client-key.pem -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/apache-client-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=ReverseProxy/CN=apache-proxy.obs-web-universe-main-dev
[2026-02-05 03:41:02 UTC] USER=www-data EUID=0 PID=3558475 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-web-universe-main-dev/apache-client-csr.pem -CA /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-web-universe-main-dev/apache-client-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = ReverseProxy, CN = apache-proxy.obs-web-universe-main-dev
[2026-02-05 03:41:02 UTC] USER=www-data EUID=0 PID=3558502 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-web-universe-main-dev/apache-client-key.pem
[2026-02-05 03:41:02 UTC] USER=www-data EUID=0 PID=3558512 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-web-universe-main-dev/apache-client-combined.pem
[2026-02-05 03:41:02 UTC] USER=www-data EUID=0 PID=3558523 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/apache-client-cert.pem
[2026-02-05 03:41:02 UTC] USER=www-data EUID=0 PID=3558534 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-web-universe-main-dev/apache-client-key.pem
[2026-02-05 03:41:02 UTC] USER=www-data EUID=0 PID=3558543 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-web-universe-main-dev/apache-client-cert.pem
[2026-02-05 03:41:02 UTC] USER=www-data EUID=0 PID=3558552 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-web-universe-main-dev/apache-client-combined.pem
[2026-02-05 03:41:02 UTC] USER=www-data EUID=0 PID=3558561 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-web-universe-main-dev/apache-client-csr.pem
[INFO] Apache client certificate created: /etc/fastorder/observability/certs/obs-web-universe-main-dev/apache-client-cert.pem
[INFO] Apache combined cert+key: /etc/fastorder/observability/certs/obs-web-universe-main-dev/apache-client-combined.pem
[INFO] Storing mTLS certificates in AWS Secrets Manager...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/web/universe/main/dev/mtls/php-client-7PIDQL",
"Name": "fastorder/observability/web/universe/main/dev/mtls/php-client",
"VersionId": "fcad1fd8-a2f3-49b5-bfef-aad737efcc13"
}
[INFO] mTLS certificates stored in Secrets Manager: fastorder/observability/web/universe/main/dev/mtls/php-client
[INFO] mTLS certificates generated successfully
[INFO] Certificate directory: /etc/fastorder/observability/certs/obs-web-universe-main-dev
[INFO] PHP client cert: /etc/fastorder/observability/certs/obs-web-universe-main-dev/php-client-cert.pem
[INFO] PHP client key: /etc/fastorder/observability/certs/obs-web-universe-main-dev/php-client-key.pem
[INFO] Apache client cert: /etc/fastorder/observability/certs/obs-web-universe-main-dev/apache-client-cert.pem
[INFO] Apache combined (for SSLProxyMachineCertificateFile): /etc/fastorder/observability/certs/obs-web-universe-main-dev/apache-client-combined.pem
[0;32m[OK][0m mTLS certificates generated
[0;34m[INFO][0m Step 5/10: Deploying log storage backend...
[0;34m[INFO][0m Provider: clickhouse (selected)
[0;34m[INFO][0m Note: Deployed before telemetry (OtelCol depends on log storage)
[0;34m[INFO][0m FQDN: logstore-web-universe-main-dev-clickhouse.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.238
[0;34m[INFO][0m Deploying log backend: clickhouse...
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m LOG STORAGE BACKEND DEPLOYMENT
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: clickhouse
[0;34m[INFO][0m Observability Cell: obs-web-universe-main-dev
[0;34m[INFO][0m FQDN: logstore-web-universe-main-dev-clickhouse.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.238
[0;34m[INFO][0m S3 Bucket: fastorder-logs-universe-dev
[0;34m[INFO][0m Retention: 90 days
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[2026-02-05 03:41:04 UTC] USER=unknown EUID=33 PID=3558634 ACTION=fsop ARGS=chmod +x /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh
/bin/chmod: changing permissions of '/opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh': Operation not permitted
[0;34m[INFO][0m Using provider: clickhouse
[0;34m[INFO][0m Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh
[0;34m[INFO][0m Executing provider deployment script...
[0;34m[INFO][0m Parsed: SERVICE=web, ZONE=universe, BRANCH=main, ENV=dev
[0;34m[INFO][0m Checking and cleaning ports before installation...
[0;34m[INFO][0m Initializing certificate directory for obs-web-universe-main-dev...
[2026-02-05 03:41:04 UTC] USER=www-data EUID=0 PID=3558651 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-02-05 03:41:04 UTC] USER=www-data EUID=0 PID=3558660 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-02-05 03:41:04 UTC] USER=www-data EUID=0 PID=3558669 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-02-05 03:41:04 UTC] USER=www-data EUID=0 PID=3558678 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-web-universe-main-dev
[0;32m[OK][0m Certificate directory initialized: /etc/fastorder/observability/certs/obs-web-universe-main-dev
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking and cleaning ports for observability cell: obs-web-universe-main-dev
[0;34m[INFO][0m IP Address: 10.100.1.238
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking for conflicting observability services...
[0;34m[INFO][0m Service clickhouse-server-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service prometheus-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service grafana-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service tempo-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service alertmanager-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Found 11 observability service(s) (all belong to current cell)
[0;34m[INFO][0m Checking for remaining processes on IP 10.100.1.238...
[0;34m[INFO][0m Scanning 15 ports...
[0;32m[OK][0m ✅ All 15 ports are FREE - ready for installation
[0;32m[OK][0m Port cleanup successful on attempt 1
[0;34m[INFO][0m Binding ClickHouse to allocated IP: 10.100.1.238
[0;34m[INFO][0m Deploying ClickHouse for obs-web-universe-main-dev
[0;34m[INFO][0m FQDN: logstore-web-universe-main-dev-clickhouse.fastorder.com
[0;34m[INFO][0m Allocated IP: 10.100.1.238
[0;34m[INFO][0m VM IP: 10.100.1.238
[0;34m[INFO][0m Ports: HTTP=8123 TCP=9000 Interserver=9009
[0;34m[INFO][0m S3 Bucket: fastorder-logs-universe-dev (region=me-central-1)
[0;34m[INFO][0m Retention: 90 days
[0;34m[INFO][0m Checking if ClickHouse is installed...
[0;32m[OK][0m ClickHouse already installed
[2026-02-05 03:41:04 UTC] USER=www-data EUID=0 PID=3558830 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-server-obs-web-universe-main-dev/config.d
[2026-02-05 03:41:04 UTC] USER=www-data EUID=0 PID=3558839 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-server-obs-web-universe-main-dev/users.d
[2026-02-05 03:41:04 UTC] USER=www-data EUID=0 PID=3558848 ACTION=fsop ARGS=mkdir -p /var/lib/clickhouse-obs-web-universe-main-dev
[2026-02-05 03:41:04 UTC] USER=www-data EUID=0 PID=3558857 ACTION=fsop ARGS=mkdir -p /var/log/clickhouse-server-obs-web-universe-main-dev
[2026-02-05 03:41:04 UTC] USER=www-data EUID=0 PID=3558866 ACTION=passthru ARGS=chmod 755 /etc/clickhouse-server-obs-web-universe-main-dev
[2026-02-05 03:41:04 UTC] USER=www-data EUID=0 PID=3558875 ACTION=passthru ARGS=chmod 700 /var/lib/clickhouse-obs-web-universe-main-dev
[2026-02-05 03:41:04 UTC] USER=www-data EUID=0 PID=3558884 ACTION=passthru ARGS=chmod 750 /var/log/clickhouse-server-obs-web-universe-main-dev
[0;34m[INFO][0m No existing logs_writer credentials found - generating new ones
[0;34m[INFO][0m No existing metrics_reader credentials found - generating new ones
[0;34m[INFO][0m TLS configuration exported for clickhouse
[0;34m[INFO][0m Cert: /etc/fastorder/observability/certs/obs-web-universe-main-dev/clickhouse-cert.pem
[0;34m[INFO][0m Key: /etc/fastorder/observability/certs/obs-web-universe-main-dev/clickhouse-key.pem
[0;34m[INFO][0m CA: /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem
[0;34m[INFO][0m Configuring certificate permissions for clickhouse (user: clickhouse)
[0;34m[INFO][0m Initializing certificate directory for obs-web-universe-main-dev...
[2026-02-05 03:41:07 UTC] USER=www-data EUID=0 PID=3558976 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-02-05 03:41:07 UTC] USER=www-data EUID=0 PID=3558985 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-02-05 03:41:07 UTC] USER=www-data EUID=0 PID=3558994 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-02-05 03:41:08 UTC] USER=www-data EUID=0 PID=3559003 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-web-universe-main-dev
[0;32m[OK][0m Certificate directory initialized: /etc/fastorder/observability/certs/obs-web-universe-main-dev
[0;34m[INFO][0m Setting file permissions...
[2026-02-05 03:41:08 UTC] USER=www-data EUID=0 PID=3559014 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/clickhouse-cert.pem
[2026-02-05 03:41:08 UTC] USER=www-data EUID=0 PID=3559023 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem
[2026-02-05 03:41:08 UTC] USER=www-data EUID=0 PID=3559033 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-web-universe-main-dev/clickhouse-key.pem
[0;34m[INFO][0m Setting file ownership...
[2026-02-05 03:41:08 UTC] USER=www-data EUID=0 PID=3559049 ACTION=passthru ARGS=chown root:clickhouse /etc/fastorder/observability/certs/obs-web-universe-main-dev/clickhouse-key.pem
[2026-02-05 03:41:08 UTC] USER=www-data EUID=0 PID=3559064 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-web-universe-main-dev/clickhouse-cert.pem /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem
[0;34m[INFO][0m Permission configuration completed
[0;34m[INFO][0m (Verification skipped - running via wrapper, trust chmod/chown success)
[0;32m[OK][0m ✅ Certificate permissions configured successfully for clickhouse
[0;34m[INFO][0m Creating ClickHouse configuration...
[2026-02-05 03:41:08 UTC] USER=www-data EUID=0 PID=3559138 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /etc/clickhouse-server-obs-web-universe-main-dev
[2026-02-05 03:41:08 UTC] USER=www-data EUID=0 PID=3559147 ACTION=passthru ARGS=bash -c chmod 640 /etc/clickhouse-server-obs-web-universe-main-dev/*.xml
[0;32m[OK][0m ClickHouse configuration created
[0;34m[INFO][0m Creating logs table schema...
[2026-02-05 03:41:08 UTC] USER=www-data EUID=0 PID=3559165 ACTION=passthru ARGS=sed -i s/__RETENTION_DAYS__/90/g /etc/clickhouse-server-obs-web-universe-main-dev/logs_schema.sql
[2026-02-05 03:41:08 UTC] USER=www-data EUID=0 PID=3559174 ACTION=passthru ARGS=chmod 644 /etc/clickhouse-server-obs-web-universe-main-dev/logs_schema.sql
[0;32m[OK][0m Logs schema created
[0;34m[INFO][0m Creating systemd service...
[2026-02-05 03:41:08 UTC] USER=www-data EUID=0 PID=3559194 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /var/lib/clickhouse-obs-web-universe-main-dev
[2026-02-05 03:41:08 UTC] USER=www-data EUID=0 PID=3559203 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /var/log/clickhouse-server-obs-web-universe-main-dev
[2026-02-05 03:41:08 UTC] USER=www-data EUID=0 PID=3559213 ACTION=passthru ARGS=chmod 700 /var/lib/clickhouse-obs-web-universe-main-dev
[0;32m[OK][0m Systemd service created
[0;34m[INFO][0m Starting ClickHouse service...
[2026-02-05 03:41:08 UTC] USER=www-data EUID=0 PID=3559222 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 03:41:09 UTC] USER=www-data EUID=0 PID=3559279 ACTION=passthru ARGS=systemctl enable clickhouse-server-obs-web-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/clickhouse-server-obs-web-universe-main-dev.service → /etc/systemd/system/clickhouse-server-obs-web-universe-main-dev.service.
[0;34m[INFO][0m Waiting for ClickHouse to be ready...
[0;32m[OK][0m ClickHouse is ready
[0;34m[INFO][0m Initializing database schema...
[0;32m[OK][0m Schema initialized
[0;34m[INFO][0m Storing ClickHouse credentials in AWS Secrets Manager...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/web/universe/main/dev/clickhouse/server/logs_writer-9wVjTM",
"Name": "fastorder/observability/web/universe/main/dev/clickhouse/server/logs_writer",
"VersionId": "94b2b174-4315-4b03-8ef2-f13141bcb019"
}
[0;32m[OK][0m logs_writer credentials stored and verified in Secrets Manager
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/web/universe/main/dev/clickhouse/server/metrics_reader-pUbGYZ",
"Name": "fastorder/observability/web/universe/main/dev/clickhouse/server/metrics_reader",
"VersionId": "60055e9d-c175-431a-8153-cad3e150b8cd"
}
[0;32m[OK][0m metrics_reader credentials stored and verified in Secrets Manager
[0;34m[INFO][0m Validating ClickHouse deployment...
[0;34m[INFO][0m ClickHouse version: 25.10.1.3832
[0;34m[INFO][0m Tables created: .inner_id.76ca8987-2518-4785-bb2a-80964de2a2ff
.inner_id.e2f024ef-c0bc-4236-84d1-8a7fd3d70fd9
application_logs
error_logs_mv
iam_audit_event
metrics_all
otel_logs
request_logs_mv
security_access
[0;34m[INFO][0m Test log inserted. Total logs: 1
[0;32m[OK][0m ✅ ClickHouse deployment validated
[0;34m[INFO][0m Setting up clickhouse-backup for backup management...
[0;32m[OK][0m clickhouse-backup already installed
[0;34m[INFO][0m Creating clickhouse-backup configuration...
[2026-02-05 03:41:20 UTC] USER=www-data EUID=0 PID=3560423 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-backup
[2026-02-05 03:41:20 UTC] USER=www-data EUID=0 PID=3560433 ACTION=passthru ARGS=chown root:clickhouse /etc/clickhouse-backup
[2026-02-05 03:41:20 UTC] USER=www-data EUID=0 PID=3560442 ACTION=passthru ARGS=chmod 750 /etc/clickhouse-backup
[2026-02-05 03:41:20 UTC] USER=www-data EUID=0 PID=3560510 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 03:41:21 UTC] USER=www-data EUID=0 PID=3560562 ACTION=passthru ARGS=systemctl enable clickhouse-backup-api-obs-web-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/clickhouse-backup-api-obs-web-universe-main-dev.service → /etc/systemd/system/clickhouse-backup-api-obs-web-universe-main-dev.service.
[2026-02-05 03:41:22 UTC] USER=www-data EUID=0 PID=3560673 ACTION=passthru ARGS=systemctl enable clickhouse-backup@obs-web-universe-main-dev.timer
Created symlink /etc/systemd/system/timers.target.wants/clickhouse-backup@obs-web-universe-main-dev.timer → /etc/systemd/system/clickhouse-backup@.timer.
[2026-02-05 03:41:22 UTC] USER=www-data EUID=0 PID=3560734 ACTION=passthru ARGS=systemctl start clickhouse-backup@obs-web-universe-main-dev.timer
[0;32m[OK][0m clickhouse-backup configured and started
[0;34m[INFO][0m Setting up ClickHouse exporter for Prometheus...
[0;32m[OK][0m clickhouse_exporter already installed
[2026-02-05 03:41:22 UTC] USER=www-data EUID=0 PID=3560769 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 03:41:23 UTC] USER=www-data EUID=0 PID=3560844 ACTION=passthru ARGS=systemctl enable clickhouse_exporter-obs-web-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/clickhouse_exporter-obs-web-universe-main-dev.service → /etc/systemd/system/clickhouse_exporter-obs-web-universe-main-dev.service.
[2026-02-05 03:41:24 UTC] USER=www-data EUID=0 PID=3560907 ACTION=passthru ARGS=systemctl start clickhouse_exporter-obs-web-universe-main-dev.service
[0;32m[OK][0m clickhouse_exporter configured and started
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ ClickHouse Deployed Successfully
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m FQDN: logstore-web-universe-main-dev-clickhouse.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.238
[0;34m[INFO][0m HTTP Port: 8123
[0;34m[INFO][0m Native Port: 9000
[0;34m[INFO][0m Database: logs
[0;34m[INFO][0m Retention: 90 days
[0;34m[INFO][0m Storage: Tiered (Local → S3: fastorder-logs-universe-dev in me-central-1)
[0;34m[INFO][0m
[0;34m[INFO][0m Backup & Monitoring:
[0;34m[INFO][0m clickhouse-backup API: http://10.100.1.238:7171
[0;34m[INFO][0m clickhouse_exporter: http://10.100.1.238:9116/metrics
[0;34m[INFO][0m Backup Schedule: Daily at 2:00 AM
[0;34m[INFO][0m Local Backups Retained: 7
[0;34m[INFO][0m
[0;34m[INFO][0m Credentials stored in AWS Secrets Manager:
[0;34m[INFO][0m Writers: fastorder/observability/web/universe/main/dev/clickhouse/server/logs_writer
[0;34m[INFO][0m Readers: fastorder/observability/web/universe/main/dev/clickhouse/server/metrics_reader (for PHP metrics service)
[0;34m[INFO][0m
[0;34m[INFO][0m Example queries (using credentials from Secrets Manager):
[0;34m[INFO][0m # Write logs:
[0;34m[INFO][0m clickhouse-client --host logstore-web-universe-main-dev-clickhouse.fastorder.com --port 9000 --user logs_writer --password '***' --query 'SELECT 1'
[0;34m[INFO][0m
[0;34m[INFO][0m # Read metrics (PHP metrics service):
[0;34m[INFO][0m clickhouse-client --host logstore-web-universe-main-dev-clickhouse.fastorder.com --port 9000 --user metrics_reader --password '***' --query 'SELECT * FROM system.metrics'
[0;34m[INFO][0m
[0;34m[INFO][0m HTTPS Setup (run on web-03/skeleton server):
[0;34m[INFO][0m # Set up HTTPS reverse proxy with Let's Encrypt:
[0;34m[INFO][0m OBS_CELL=obs-web-universe-main-dev BACKEND_IP=10.100.1.238 sudo bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/../https/setup-clickhouse-https.sh
[0;34m[INFO][0m
[0;34m[INFO][0m # Or add --setup-https flag when running this script
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Log Storage Backend Deployed Successfully
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: clickhouse
[0;34m[INFO][0m FQDN: logstore-web-universe-main-dev-clickhouse.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.238
[0;34m[INFO][0m Retention: 90 days
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Registering ClickHouse in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: ClickHouse
[INFO] Identifier: web-universe-main-dev-clickhouse
[INFO] Identifier Parent: cluster
[INFO] IP: 10.100.1.238
[INFO] Port: 8443
[INFO] FQDN: logstore-web-universe-main-dev-clickhouse.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ INVALID REQUEST
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] Response: {"success":false,"error":"Invalid JSON: Control character error, possibly incorrectly encoded"}
[ERROR]
[ERROR] Request payload:
{
"env_id": "web-universe-main-dev",
"application": "ClickHouse",
"identifier": "web-universe-main-dev-clickhouse",
"identifier_parent": "cluster",
"ip": "10.100.1.238",
"port": 8443,
"fqdn": "logstore-web-universe-main-dev-clickhouse.fastorder.com",
"status": "running",
"meta": {
"role": "log_storage",
"provider": "clickhouse",
"version": "25.10
1.3832",
"http_port": 8123,
"native_port": 9000,
"https_port": 8443,
"protocol": "https",
"metrics_enabled": true,
"metrics_port": 8123,
"metrics_path": "/metrics",
"health_endpoint": "https://logstore-web-universe-main-dev-clickhouse.fastorder.com/ping",
"retention_days": 90,
"s3_bucket": "fastorder-logs-universe-dev"
}
}
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[1;33m[WARN][0m ⚠️ Failed to register ClickHouse (service is running)
[0;32m[OK][0m clickhouse deployed successfully
[0;32m[OK][0m Log storage backend deployed
[0;34m[INFO][0m Step 6/10: Deploying telemetry collector...
[0;34m[INFO][0m Provider: otlp (backend implementation - internal)
[0;34m[INFO][0m Endpoint: telemetry-web-universe-main-dev-opentelemetry.fastorder.com (stable, exposed to clients)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m TELEMETRY COLLECTOR DEPLOYMENT
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: otlp
[0;34m[INFO][0m Observability Cell: obs-web-universe-main-dev
[0;34m[INFO][0m FQDN: telemetry-web-universe-main-dev-opentelemetry.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.241
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Using provider: otlp
[0;34m[INFO][0m Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Telemetry/provider/otlp.sh
[0;34m[INFO][0m Executing provider deployment script...
[0;34m[INFO][0m Parsed: SERVICE=web, ZONE=universe, BRANCH=main, ENV=dev
[0;34m[INFO][0m Checking and cleaning ports before installation...
[0;34m[INFO][0m Initializing certificate directory for obs-web-universe-main-dev...
[2026-02-05 03:41:24 UTC] USER=www-data EUID=0 PID=3560965 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-02-05 03:41:24 UTC] USER=www-data EUID=0 PID=3560974 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-02-05 03:41:24 UTC] USER=www-data EUID=0 PID=3560983 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-02-05 03:41:24 UTC] USER=www-data EUID=0 PID=3560992 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-web-universe-main-dev
[0;32m[OK][0m Certificate directory initialized: /etc/fastorder/observability/certs/obs-web-universe-main-dev
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking and cleaning ports for observability cell: obs-web-universe-main-dev
[0;34m[INFO][0m IP Address: 10.100.1.241
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking for conflicting observability services...
[0;34m[INFO][0m Service clickhouse-server-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service clickhouse-server-obs-web-universe-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Service clickhouse-server@obs-web-universe-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service prometheus-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service grafana-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service tempo-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service alertmanager-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Found 13 observability service(s) (all belong to current cell)
[0;34m[INFO][0m Checking for remaining processes on IP 10.100.1.241...
[0;34m[INFO][0m Scanning 15 ports...
[0;32m[OK][0m ✅ All 15 ports are FREE - ready for installation
[0;32m[OK][0m Port cleanup successful on attempt 1
[0;34m[INFO][0m Binding to allocated IP: 10.100.1.241
[0;34m[INFO][0m Deploying OpenTelemetry Collector for observability cell: obs-web-universe-main-dev
[0;34m[INFO][0m FQDN: telemetry-web-universe-main-dev-opentelemetry.fastorder.com
[0;34m[INFO][0m Allocated IP: 10.100.1.241
[0;34m[INFO][0m VM IP: 10.100.1.241
[0;34m[INFO][0m Ports: gRPC=4317 HTTP=4318 Metrics=8888 Prom=8889
[0;34m[VERSION][0m Fetching latest version for otel_collector from GitHub (open-telemetry/opentelemetry-collector-releases)...
[0;32m[VERSION][0m Latest otel_collector version: 0.144.0
[0;34m[INFO][0m Resolved OpenTelemetry Collector version: 0.144.0
[0;32m[OK][0m User 'otelcol' already exists
[0;34m[INFO][0m Checking if OpenTelemetry Collector is installed...
[0;32m[OK][0m OpenTelemetry Collector already installed at /usr/local/bin/otelcol-contrib
[0;34m[INFO][0m Creating configuration/data directories...
[2026-02-05 03:41:26 UTC] USER=www-data EUID=0 PID=3561187 ACTION=passthru ARGS=mkdir -p /etc/otelcol/obs-web-universe-main-dev
[2026-02-05 03:41:26 UTC] USER=www-data EUID=0 PID=3561206 ACTION=passthru ARGS=mkdir -p /var/lib/otelcol/obs-web-universe-main-dev
[2026-02-05 03:41:26 UTC] USER=www-data EUID=0 PID=3561215 ACTION=passthru ARGS=chown -R otelcol:otelcol /etc/otelcol/obs-web-universe-main-dev /var/lib/otelcol/obs-web-universe-main-dev
[2026-02-05 03:41:26 UTC] USER=www-data EUID=0 PID=3561224 ACTION=passthru ARGS=chmod 0750 /etc/otelcol/obs-web-universe-main-dev
[2026-02-05 03:41:26 UTC] USER=www-data EUID=0 PID=3561233 ACTION=passthru ARGS=chmod 0750 /var/lib/otelcol/obs-web-universe-main-dev
[0;34m[INFO][0m Retrieving ClickHouse credentials from Secrets Manager...
[0;32m[OK][0m Retrieved ClickHouse credentials from Secrets Manager
[0;34m[INFO][0m Creating OpenTelemetry Collector configuration...
[0;34m[INFO][0m ClickHouse exporter enabled: tcp://logstore-web-universe-main-dev-clickhouse.fastorder.com:9000
[2026-02-05 03:41:28 UTC] USER=www-data EUID=0 PID=3561312 ACTION=passthru ARGS=chown otelcol:otelcol /etc/otelcol/obs-web-universe-main-dev/config.yaml
[2026-02-05 03:41:28 UTC] USER=www-data EUID=0 PID=3561321 ACTION=passthru ARGS=chmod 0640 /etc/otelcol/obs-web-universe-main-dev/config.yaml
[0;32m[OK][0m Configuration created at /etc/otelcol/obs-web-universe-main-dev/config.yaml
[0;34m[INFO][0m Setting up TLS certificate permissions...
[0;34m[INFO][0m Configuring certificate permissions for otlp_collector (user: otelcol)
[0;34m[INFO][0m Initializing certificate directory for obs-web-universe-main-dev...
[2026-02-05 03:41:28 UTC] USER=www-data EUID=0 PID=3561330 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-02-05 03:41:28 UTC] USER=www-data EUID=0 PID=3561348 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-02-05 03:41:28 UTC] USER=www-data EUID=0 PID=3561357 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-web-universe-main-dev
[0;32m[OK][0m Certificate directory initialized: /etc/fastorder/observability/certs/obs-web-universe-main-dev
[0;34m[INFO][0m Setting file permissions...
[2026-02-05 03:41:28 UTC] USER=www-data EUID=0 PID=3561367 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/otlp_collector-cert.pem
[2026-02-05 03:41:28 UTC] USER=www-data EUID=0 PID=3561381 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem
[2026-02-05 03:41:28 UTC] USER=www-data EUID=0 PID=3561403 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-web-universe-main-dev/otlp_collector-key.pem
[0;34m[INFO][0m Setting file ownership...
[2026-02-05 03:41:28 UTC] USER=www-data EUID=0 PID=3561420 ACTION=passthru ARGS=chown root:otelcol /etc/fastorder/observability/certs/obs-web-universe-main-dev/otlp_collector-key.pem
[2026-02-05 03:41:28 UTC] USER=www-data EUID=0 PID=3561435 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-web-universe-main-dev/otlp_collector-cert.pem /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem
[0;34m[INFO][0m Permission configuration completed
[0;34m[INFO][0m (Verification skipped - running via wrapper, trust chmod/chown success)
[0;32m[OK][0m ✅ Certificate permissions configured successfully for otlp_collector
[0;32m[OK][0m Certificate permissions configured
[0;34m[INFO][0m Creating systemd service: otelcol-obs-web-universe-main-dev
[0;32m[OK][0m Systemd service created at /etc/systemd/system/otelcol-obs-web-universe-main-dev.service
[0;34m[INFO][0m Adding /etc/hosts entry for telemetry-web-universe-main-dev-opentelemetry.fastorder.com -> 10.100.1.241
[2026-02-05 03:41:29 UTC] USER=www-data EUID=0 PID=3561459 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*telemetry-web-universe-main-dev-opentelemetry.fastorder.com/10.100.1.241 telemetry-web-universe-main-dev-opentelemetry.fastorder.com/ /etc/hosts
[0;32m[OK][0m Updated /etc/hosts entry to use VM_IP
[0;34m[INFO][0m Storing OTLP configuration metadata in AWS Secrets Manager (if aws CLI present)...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/web/universe/main/dev/otlp/collector-bkOLah",
"Name": "fastorder/observability/web/universe/main/dev/otlp/collector",
"VersionId": "5d8817f3-c764-405a-8dd7-7b64eef907f1"
}
[0;32m[OK][0m Configuration metadata stored/updated in AWS Secrets Manager: fastorder/observability/web/universe/main/dev/otlp/collector
[0;34m[INFO][0m Enabling and starting OpenTelemetry Collector service...
[2026-02-05 03:41:30 UTC] USER=www-data EUID=0 PID=3561506 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 03:41:31 UTC] USER=www-data EUID=0 PID=3561560 ACTION=passthru ARGS=systemctl enable otelcol-obs-web-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/otelcol-obs-web-universe-main-dev.service → /etc/systemd/system/otelcol-obs-web-universe-main-dev.service.
[2026-02-05 03:41:32 UTC] USER=www-data EUID=0 PID=3561617 ACTION=passthru ARGS=systemctl restart otelcol-obs-web-universe-main-dev.service
[0;32m[OK][0m Service enabled and started
[0;34m[INFO][0m Validating deployment...
[2026-02-05 03:41:35 UTC] USER=www-data EUID=0 PID=3561769 ACTION=passthru ARGS=systemctl is-active --quiet otelcol-obs-web-universe-main-dev.service
[0;32m[OK][0m ✅ OpenTelemetry Collector is running
[0;32m[OK][0m ✅ gRPC endpoint listening on port 4317
[0;32m[OK][0m ✅ HTTP endpoint listening on port 4318
[0;32m[OK][0m ✅ Prometheus metrics endpoint listening on port 8889
[0;34m[INFO][0m Service logs (last 10 lines):
[2026-02-05 03:41:35 UTC] USER=www-data EUID=0 PID=3561787 ACTION=passthru ARGS=journalctl -u otelcol-obs-web-universe-main-dev.service -n 10 --no-pager
Feb 05 03:41:32 web-03 otelcol-obs-web-universe-main-dev[3561629]: 2026-02-05T03:41:32.648Z info internal/resourcedetection.go:125 began detecting resource information {"kind": "processor", "name": "resourcedetection", "pipeline": "traces"}
Feb 05 03:41:32 web-03 otelcol-obs-web-universe-main-dev[3561629]: 2026-02-05T03:41:32.650Z info system/system.go:201 This attribute changed from int to string. Temporarily switch back to int using the feature gate. {"kind": "processor", "name": "resourcedetection", "pipeline": "traces", "attribute": "host.cpu.family", "feature gate": "processor.resourcedetection.hostCPUModelAndFamilyAsString"}
Feb 05 03:41:32 web-03 otelcol-obs-web-universe-main-dev[3561629]: 2026-02-05T03:41:32.651Z info system/system.go:220 This attribute changed from int to string. Temporarily switch back to int using the feature gate. {"kind": "processor", "name": "resourcedetection", "pipeline": "traces", "attribute": "host.cpu.model.id", "feature gate": "processor.resourcedetection.hostCPUModelAndFamilyAsString"}
Feb 05 03:41:32 web-03 otelcol-obs-web-universe-main-dev[3561629]: 2026-02-05T03:41:32.652Z info internal/resourcedetection.go:139 detected resource information {"kind": "processor", "name": "resourcedetection", "pipeline": "traces", "resource": {"host.name":"web-03","os.type":"linux"}}
Feb 05 03:41:32 web-03 otelcol-obs-web-universe-main-dev[3561629]: 2026-02-05T03:41:32.702Z info otlpreceiver@v0.91.0/otlp.go:83 Starting GRPC server {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "10.100.1.241:4317"}
Feb 05 03:41:32 web-03 otelcol-obs-web-universe-main-dev[3561629]: 2026-02-05T03:41:32.704Z info otlpreceiver@v0.91.0/otlp.go:101 Starting HTTP server {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "10.100.1.241:4318"}
Feb 05 03:41:32 web-03 otelcol-obs-web-universe-main-dev[3561629]: 2026-02-05T03:41:32.705Z info prometheusreceiver@v0.91.0/metrics_receiver.go:231 Scrape job added {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "jobName": "otel-collector"}
Feb 05 03:41:32 web-03 otelcol-obs-web-universe-main-dev[3561629]: 2026-02-05T03:41:32.705Z info prometheusreceiver@v0.91.0/metrics_receiver.go:240 Starting discovery manager {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
Feb 05 03:41:32 web-03 otelcol-obs-web-universe-main-dev[3561629]: 2026-02-05T03:41:32.706Z info service@v0.91.0/service.go:171 Everything is ready. Begin running and processing data.
Feb 05 03:41:32 web-03 otelcol-obs-web-universe-main-dev[3561629]: 2026-02-05T03:41:32.706Z info prometheusreceiver@v0.91.0/metrics_receiver.go:282 Starting scrape manager {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Telemetry Collector Deployed Successfully
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: otlp
[0;34m[INFO][0m FQDN: telemetry-web-universe-main-dev-opentelemetry.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.241
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Registering OpenTelemetry Collector in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: OpenTelemetry Collector
[INFO] Identifier: web-universe-main-dev-opentelemetry
[INFO] Identifier Parent: cluster
[INFO] IP: 10.100.1.241
[INFO] Port: 4317
[INFO] FQDN: telemetry-web-universe-main-dev-opentelemetry.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[WARN] Registration API call failed (HTTP 500), retrying (1/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[WARN] Registration API call failed (HTTP 500), retrying (2/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ REGISTRATION FAILED AFTER 3 ATTEMPTS
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] HTTP Status: 500
[ERROR] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[ERROR]
[ERROR] API endpoint: https://skeleton.dev.fastorder.com/api/obs/register
[ERROR]
[ERROR] Troubleshooting:
[ERROR] 1. Check if skeleton.dev.fastorder.com is accessible
[ERROR] 2. Verify web application is running
[ERROR] 3. Check web application logs: /var/www/html/skeleton.dev.fastorder.com/logs/
[ERROR] 4. Test API manually:
[ERROR] curl -k -X POST 'https://skeleton.dev.fastorder.com/api/obs/register' \
[ERROR] -H 'Content-Type: application/json' \
[ERROR] -H 'X-Internal-Token: $OBS_INTERNAL_API_TOKEN' \
[ERROR] -d '$PAYLOAD'
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[1;33m[WARN][0m ⚠️ Failed to register OpenTelemetry Collector (service is running)
[0;34m[INFO][0m Setting up OpenTelemetry Collector metrics collection timer...
[2026-02-05 03:41:41 UTC] USER=www-data EUID=0 PID=3561981 ACTION=passthru ARGS=mv /tmp/otelcol-metrics-web-universe-main-dev.service /etc/systemd/system/
[2026-02-05 03:41:41 UTC] USER=www-data EUID=0 PID=3561991 ACTION=passthru ARGS=mv /tmp/otelcol-metrics-web-universe-main-dev.timer /etc/systemd/system/
[2026-02-05 03:41:41 UTC] USER=www-data EUID=0 PID=3562000 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 03:41:41 UTC] USER=www-data EUID=0 PID=3562049 ACTION=passthru ARGS=systemctl enable otelcol-metrics-web-universe-main-dev.timer
Created symlink /etc/systemd/system/timers.target.wants/otelcol-metrics-web-universe-main-dev.timer → /etc/systemd/system/otelcol-metrics-web-universe-main-dev.timer.
[2026-02-05 03:41:42 UTC] USER=www-data EUID=0 PID=3562098 ACTION=passthru ARGS=systemctl start otelcol-metrics-web-universe-main-dev.timer
[0;32m[OK][0m ✅ Metrics collection timer installed and started
[0;32m[OK][0m Telemetry collector (otlp) deployed successfully
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Step 7/10: METRICS BACKEND DEPLOYMENT
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Provider: prometheus
[0;34m[INFO][0m OBS Cell: obs-web-universe-main-dev
[0;34m[INFO][0m FQDN: metrics-web-universe-main-dev-prometheus.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.236
[0;34m[INFO][0m Script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/deploy-metrics.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 📊 METRICS DEPLOYMENT WRAPPER STARTED
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Script: deploy-metrics.sh
[0;34m[INFO][0m Timestamp: 2026-02-05 03:41:42 UTC
[0;34m[INFO][0m Arguments: --provider prometheus --obs-cell obs-web-universe-main-dev --fqdn metrics-web-universe-main-dev-prometheus.fastorder.com --ip 10.100.1.236
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m METRICS DEPLOYMENT
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: prometheus
[0;34m[INFO][0m Observability Cell: obs-web-universe-main-dev
[0;34m[INFO][0m FQDN: metrics-web-universe-main-dev-prometheus.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.236
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Using provider: prometheus
[0;34m[INFO][0m Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/provider/prometheus.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/provider/prometheus.sh
[0;34m[INFO][0m OBS_CELL: obs-web-universe-main-dev
[0;34m[INFO][0m FQDN: metrics-web-universe-main-dev-prometheus.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.236
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Parsed: SERVICE=web, ZONE=universe, BRANCH=main, ENV=dev
[0;34m[INFO][0m Checking and cleaning ports before installation...
[0;34m[INFO][0m Initializing certificate directory for obs-web-universe-main-dev...
[2026-02-05 03:41:42 UTC] USER=www-data EUID=0 PID=3562143 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-02-05 03:41:42 UTC] USER=www-data EUID=0 PID=3562168 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-02-05 03:41:42 UTC] USER=www-data EUID=0 PID=3562198 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-02-05 03:41:42 UTC] USER=www-data EUID=0 PID=3562240 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-web-universe-main-dev
[0;32m[OK][0m Certificate directory initialized: /etc/fastorder/observability/certs/obs-web-universe-main-dev
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking and cleaning ports for observability cell: obs-web-universe-main-dev
[0;34m[INFO][0m IP Address: 10.100.1.236
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking for conflicting observability services...
[0;34m[INFO][0m Service clickhouse-server-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service clickhouse-server-obs-web-universe-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Service clickhouse-server@obs-web-universe-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-obs-web-universe-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Service prometheus-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service grafana-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service tempo-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service alertmanager-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Found 15 observability service(s) (all belong to current cell)
[0;34m[INFO][0m Checking for remaining processes on IP 10.100.1.236...
[0;34m[INFO][0m Scanning 15 ports...
[0;32m[OK][0m ✅ All 15 ports are FREE - ready for installation
[0;32m[OK][0m Port cleanup successful on attempt 1
[0;34m[INFO][0m Deploying Prometheus for observability cell: obs-web-universe-main-dev
[0;34m[INFO][0m FQDN: metrics-web-universe-main-dev-prometheus.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.236
[0;34m[INFO][0m Prometheus Port: 9090
[0;34m[VERSION][0m Fetching latest version for prometheus from GitHub (prometheus/prometheus)...
[0;32m[VERSION][0m Latest prometheus version: 3.9.1
[0;34m[INFO][0m Resolved Prometheus version: 3.9.1
[0;34m[INFO][0m Checking if Prometheus is installed...
[0;32m[OK][0m Prometheus already installed at /usr/local/bin/prometheus
[0;34m[VERSION][0m Fetching latest version for node_exporter from GitHub (prometheus/node_exporter)...
[0;32m[VERSION][0m Latest node_exporter version: 1.10.2
[0;34m[INFO][0m Resolved Node Exporter version: 1.10.2
[0;34m[INFO][0m Checking if Node Exporter is installed...
[0;32m[OK][0m Node Exporter already installed at /usr/local/bin/node_exporter
[0;34m[INFO][0m Creating Node Exporter TLS web config...
[0;34m[INFO][0m Creating Node Exporter systemd service with TLS...
[2026-02-05 03:41:44 UTC] USER=www-data EUID=0 PID=3562545 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 03:41:44 UTC] USER=www-data EUID=0 PID=3562611 ACTION=passthru ARGS=systemctl enable node_exporter-obs-web-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/node_exporter-obs-web-universe-main-dev.service → /etc/systemd/system/node_exporter-obs-web-universe-main-dev.service.
[2026-02-05 03:41:45 UTC] USER=www-data EUID=0 PID=3562676 ACTION=passthru ARGS=systemctl restart node_exporter-obs-web-universe-main-dev.service
[0;32m[OK][0m Node Exporter service configured and started
[0;34m[INFO][0m Creating configuration directory: /etc/prometheus/obs-web-universe-main-dev
[2026-02-05 03:41:45 UTC] USER=www-data EUID=0 PID=3562690 ACTION=fsop ARGS=mkdir -p /etc/prometheus/obs-web-universe-main-dev
[2026-02-05 03:41:45 UTC] USER=www-data EUID=0 PID=3562699 ACTION=fsop ARGS=mkdir -p /var/lib/prometheus/obs-web-universe-main-dev
[2026-02-05 03:41:45 UTC] USER=www-data EUID=0 PID=3562708 ACTION=fsop ARGS=mkdir -p /etc/prometheus/obs-web-universe-main-dev/rules
[0;34m[INFO][0m Creating Prometheus configuration...
[0;34m[INFO][0m Generated FQDNs:
[0;34m[INFO][0m Prometheus: metrics-web-universe-main-dev-prometheus.fastorder.com
[0;34m[INFO][0m Alertmanager: alerts-web-universe-main-dev-alertmanager.fastorder.com
[0;34m[INFO][0m Grafana: dashboards-web-universe-main-dev-grafana.fastorder.com
[0;34m[INFO][0m Otelcol: telemetry-web-universe-main-dev-opentelemetry.fastorder.com
[0;32m[OK][0m Configuration created at /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
[0;34m[INFO][0m Creating Prometheus web config for HTTPS...
[0;32m[OK][0m Web config created at /etc/prometheus/obs-web-universe-main-dev/web-config.yml
[0;34m[INFO][0m Creating basic alerting rules...
[0;32m[OK][0m Alerting rules created
[2026-02-05 03:41:45 UTC] USER=www-data EUID=0 PID=3562745 ACTION=fsop ARGS=mkdir -p /etc/prometheus/obs-web-universe-main-dev/targets
[2026-02-05 03:41:45 UTC] USER=www-data EUID=0 PID=3562756 ACTION=passthru ARGS=bash -c cat > '/etc/prometheus/obs-web-universe-main-dev/targets/.placeholder.yml' << 'EOF'
# Placeholder file to prevent file_sd_configs warning
# Application targets will be added here automatically
[]
EOF
[0;34m[INFO][0m Creating systemd service: prometheus-obs-web-universe-main-dev
[0;34m[INFO][0m Binding to: 10.100.1.236:9090
[0;32m[OK][0m Systemd service created
[0;34m[INFO][0m Configuring certificate permissions...
[0;34m[INFO][0m Configuring certificate permissions for prometheus (user: root)
[0;34m[INFO][0m Initializing certificate directory for obs-web-universe-main-dev...
[2026-02-05 03:41:45 UTC] USER=www-data EUID=0 PID=3562775 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-02-05 03:41:45 UTC] USER=www-data EUID=0 PID=3562784 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-02-05 03:41:45 UTC] USER=www-data EUID=0 PID=3562793 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-02-05 03:41:45 UTC] USER=www-data EUID=0 PID=3562802 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-web-universe-main-dev
[0;32m[OK][0m Certificate directory initialized: /etc/fastorder/observability/certs/obs-web-universe-main-dev
[0;34m[INFO][0m Setting file permissions...
[2026-02-05 03:41:45 UTC] USER=www-data EUID=0 PID=3562812 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/prometheus-cert.pem
[2026-02-05 03:41:45 UTC] USER=www-data EUID=0 PID=3562821 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem
[2026-02-05 03:41:45 UTC] USER=www-data EUID=0 PID=3562830 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-web-universe-main-dev/prometheus-key.pem
[0;34m[INFO][0m Setting file ownership...
[2026-02-05 03:41:45 UTC] USER=www-data EUID=0 PID=3562839 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-web-universe-main-dev/prometheus-key.pem
[2026-02-05 03:41:45 UTC] USER=www-data EUID=0 PID=3562848 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-web-universe-main-dev/prometheus-cert.pem /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem
[0;34m[INFO][0m Permission configuration completed
[0;34m[INFO][0m (Verification skipped - running via wrapper, trust chmod/chown success)
[0;32m[OK][0m ✅ Certificate permissions configured successfully for prometheus
[0;32m[OK][0m Certificate permissions configured
[0;34m[INFO][0m Adding /etc/hosts entry for metrics-web-universe-main-dev-prometheus.fastorder.com -> 10.100.1.236
[2026-02-05 03:41:45 UTC] USER=www-data EUID=0 PID=3562859 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*metrics-web-universe-main-dev-prometheus.fastorder.com/10.100.1.236 metrics-web-universe-main-dev-prometheus.fastorder.com/ /etc/hosts
[0;32m[OK][0m Updated /etc/hosts entry to use VM_IP
[0;34m[INFO][0m Validating Prometheus configuration...
Checking /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
SUCCESS: 1 rule files found
SUCCESS: /etc/prometheus/obs-web-universe-main-dev/prometheus.yml is valid prometheus config file syntax
Checking /etc/prometheus/obs-web-universe-main-dev/rules/basic_alerts.yml
SUCCESS: 4 rules found
[0;32m[OK][0m ✅ Configuration is valid
[0;34m[INFO][0m Storing Prometheus configuration in AWS Secrets Manager...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/web/universe/main/dev/prometheus/server-yo3CTX",
"Name": "fastorder/observability/web/universe/main/dev/prometheus/server",
"VersionId": "f70bbdb3-dfd9-43cd-940e-d6c023994596"
}
[0;32m[OK][0m Configuration stored in AWS Secrets Manager
[0;34m[INFO][0m Enabling and starting Prometheus service...
[2026-02-05 03:41:47 UTC] USER=www-data EUID=0 PID=3562904 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 03:41:47 UTC] USER=www-data EUID=0 PID=3562960 ACTION=passthru ARGS=systemctl enable prometheus-obs-web-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus-obs-web-universe-main-dev.service → /etc/systemd/system/prometheus-obs-web-universe-main-dev.service.
[2026-02-05 03:41:48 UTC] USER=www-data EUID=0 PID=3563028 ACTION=passthru ARGS=systemctl restart prometheus-obs-web-universe-main-dev.service
[0;32m[OK][0m Service enabled and started
[0;34m[INFO][0m Validating deployment...
[2026-02-05 03:41:51 UTC] USER=www-data EUID=0 PID=3563090 ACTION=passthru ARGS=systemctl is-active --quiet prometheus-obs-web-universe-main-dev.service
[0;32m[OK][0m ✅ Prometheus is running
[0;32m[OK][0m ✅ Prometheus web interface listening on port 9090
[0;32m[OK][0m ✅ Prometheus health check passed (HTTPS)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m Prometheus Web UI: https://metrics-web-universe-main-dev-prometheus.fastorder.com:9090
[0;32m[OK][0m Targets: https://metrics-web-universe-main-dev-prometheus.fastorder.com:9090/targets
[0;32m[OK][0m Alerts: https://metrics-web-universe-main-dev-prometheus.fastorder.com:9090/alerts
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Service logs (last 10 lines):
[2026-02-05 03:41:53 UTC] USER=www-data EUID=0 PID=3563213 ACTION=passthru ARGS=journalctl -u prometheus-obs-web-universe-main-dev.service -n 10 --no-pager
Feb 05 03:41:48 web-03 prometheus-obs-web-universe-main-dev[3563035]: ts=2026-02-05T03:41:48.588Z caller=head.go:722 level=info component=tsdb msg="Replaying WAL, this may take a while"
Feb 05 03:41:48 web-03 prometheus-obs-web-universe-main-dev[3563035]: ts=2026-02-05T03:41:48.590Z caller=head.go:794 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
Feb 05 03:41:48 web-03 prometheus-obs-web-universe-main-dev[3563035]: ts=2026-02-05T03:41:48.590Z caller=head.go:831 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=659.34µs wal_replay_duration=1.113485ms wbl_replay_duration=300ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=2.695µs total_replay_duration=1.802702ms
Feb 05 03:41:48 web-03 prometheus-obs-web-universe-main-dev[3563035]: ts=2026-02-05T03:41:48.593Z caller=main.go:1218 level=info fs_type=EXT4_SUPER_MAGIC
Feb 05 03:41:48 web-03 prometheus-obs-web-universe-main-dev[3563035]: ts=2026-02-05T03:41:48.593Z caller=main.go:1221 level=info msg="TSDB started"
Feb 05 03:41:48 web-03 prometheus-obs-web-universe-main-dev[3563035]: ts=2026-02-05T03:41:48.593Z caller=main.go:1404 level=info msg="Loading configuration file" filename=/etc/prometheus/obs-web-universe-main-dev/prometheus.yml
Feb 05 03:41:48 web-03 prometheus-obs-web-universe-main-dev[3563035]: ts=2026-02-05T03:41:48.597Z caller=main.go:1441 level=info msg="updated GOGC" old=100 new=75
Feb 05 03:41:48 web-03 prometheus-obs-web-universe-main-dev[3563035]: ts=2026-02-05T03:41:48.597Z caller=main.go:1452 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/obs-web-universe-main-dev/prometheus.yml totalDuration=4.044903ms db_storage=2.194µs remote_storage=2.445µs web_handler=1.022µs query_engine=1.933µs scrape=825.135µs scrape_sd=110.171µs notify=37.541µs notify_sd=7.915µs rules=1.635102ms tracing=10.36µs
Feb 05 03:41:48 web-03 prometheus-obs-web-universe-main-dev[3563035]: ts=2026-02-05T03:41:48.597Z caller=main.go:1182 level=info msg="Server is ready to receive web requests."
Feb 05 03:41:48 web-03 prometheus-obs-web-universe-main-dev[3563035]: ts=2026-02-05T03:41:48.597Z caller=manager.go:164 level=info component="rule manager" msg="Starting rule manager..."
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Provider script completed with exit code: 0
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Metrics Deployed Successfully
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: prometheus
[0;34m[INFO][0m FQDN: metrics-web-universe-main-dev-prometheus.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.236
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Registering Prometheus in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Prometheus
[INFO] Identifier: web-universe-main-dev-prometheus
[INFO] Identifier Parent: cluster
[INFO] IP: 10.100.1.236
[INFO] Port: 9090
[INFO] FQDN: metrics-web-universe-main-dev-prometheus.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[WARN] Registration API call failed (HTTP 500), retrying (1/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[WARN] Registration API call failed (HTTP 500), retrying (2/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ REGISTRATION FAILED AFTER 3 ATTEMPTS
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] HTTP Status: 500
[ERROR] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[ERROR]
[ERROR] API endpoint: https://skeleton.dev.fastorder.com/api/obs/register
[ERROR]
[ERROR] Troubleshooting:
[ERROR] 1. Check if skeleton.dev.fastorder.com is accessible
[ERROR] 2. Verify web application is running
[ERROR] 3. Check web application logs: /var/www/html/skeleton.dev.fastorder.com/logs/
[ERROR] 4. Test API manually:
[ERROR] curl -k -X POST 'https://skeleton.dev.fastorder.com/api/obs/register' \
[ERROR] -H 'Content-Type: application/json' \
[ERROR] -H 'X-Internal-Token: $OBS_INTERNAL_API_TOKEN' \
[ERROR] -d '$PAYLOAD'
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[1;33m[WARN][0m Failed to register Prometheus (service is running)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Step 7/10: METRICS DEPLOYMENT RESULT
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Exit code: 0
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Metrics backend (prometheus) deployed successfully
[0;34m[INFO][0m Step 8/10: Deploying traces backend...
[0;34m[INFO][0m Provider: tempo (selected)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m TRACES DEPLOYMENT
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: tempo
[0;34m[INFO][0m Observability Cell: obs-web-universe-main-dev
[0;34m[INFO][0m FQDN: traces-web-universe-main-dev-tempo.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.239
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Using provider: tempo
[0;34m[INFO][0m Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Traces/provider/tempo.sh
[0;34m[INFO][0m Executing provider deployment script...
[0;34m[INFO][0m Parsed: SERVICE=web, ZONE=universe, BRANCH=main, ENV=dev
[0;34m[INFO][0m Checking and cleaning ports before installation...
[0;34m[INFO][0m Initializing certificate directory for obs-web-universe-main-dev...
[2026-02-05 03:41:59 UTC] USER=www-data EUID=0 PID=3563486 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-02-05 03:41:59 UTC] USER=www-data EUID=0 PID=3563495 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-02-05 03:41:59 UTC] USER=www-data EUID=0 PID=3563505 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-02-05 03:41:59 UTC] USER=www-data EUID=0 PID=3563514 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-web-universe-main-dev
[0;32m[OK][0m Certificate directory initialized: /etc/fastorder/observability/certs/obs-web-universe-main-dev
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking and cleaning ports for observability cell: obs-web-universe-main-dev
[0;34m[INFO][0m IP Address: 10.100.1.239
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking for conflicting observability services...
[0;34m[INFO][0m Service clickhouse-server-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service clickhouse-server-obs-web-universe-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Service clickhouse-server@obs-web-universe-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-web-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-obs-web-universe-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Service prometheus-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service prometheus-obs-web-universe-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Service grafana-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service tempo-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service alertmanager-obs-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Found 16 observability service(s) (all belong to current cell)
[0;34m[INFO][0m Checking for remaining processes on IP 10.100.1.239...
[0;34m[INFO][0m Scanning 15 ports...
[0;32m[OK][0m ✅ All 15 ports are FREE - ready for installation
[0;32m[OK][0m Port cleanup successful on attempt 1
[0;34m[INFO][0m Binding Tempo to allocated IP: 10.100.1.239
[0;34m[INFO][0m Deploying Grafana Tempo for observability cell: obs-web-universe-main-dev
[0;34m[INFO][0m FQDN: traces-web-universe-main-dev-tempo.fastorder.com
[0;34m[INFO][0m Allocated IP: 10.100.1.239
[0;34m[INFO][0m VM IP: 10.100.1.239
[0;34m[INFO][0m Ports: HTTP=3200 gRPC=9334, OTLP gRPC=4317, OTLP HTTP=4318
[0;34m[VERSION][0m Fetching latest version for tempo from GitHub (grafana/tempo)...
[0;32m[VERSION][0m Latest tempo version: 2.10.0
[0;34m[INFO][0m Resolved Tempo version: 2.10.0
[0;34m[INFO][0m Checking if Grafana Tempo is installed...
[0;32m[OK][0m Grafana Tempo already installed at /usr/local/bin/tempo
[0;34m[INFO][0m Preparing configuration and data directories...
[2026-02-05 03:42:00 UTC] USER=www-data EUID=0 PID=3563713 ACTION=passthru ARGS=mkdir -p /etc/tempo/obs-web-universe-main-dev
[2026-02-05 03:42:00 UTC] USER=www-data EUID=0 PID=3563722 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-web-universe-main-dev
[2026-02-05 03:42:00 UTC] USER=www-data EUID=0 PID=3563731 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-web-universe-main-dev/wal
[2026-02-05 03:42:00 UTC] USER=www-data EUID=0 PID=3563749 ACTION=passthru ARGS=chown -R tempo:tempo /etc/tempo/obs-web-universe-main-dev /var/lib/tempo/obs-web-universe-main-dev
[2026-02-05 03:42:00 UTC] USER=www-data EUID=0 PID=3563758 ACTION=passthru ARGS=chmod 750 /etc/tempo/obs-web-universe-main-dev /var/lib/tempo/obs-web-universe-main-dev
[0;34m[INFO][0m Creating Grafana Tempo configuration...
[0;34m[INFO][0m TLS configuration exported for tempo
[0;34m[INFO][0m Cert: /etc/fastorder/observability/certs/obs-web-universe-main-dev/tempo-cert.pem
[0;34m[INFO][0m Key: /etc/fastorder/observability/certs/obs-web-universe-main-dev/tempo-key.pem
[0;34m[INFO][0m CA: /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem
[0;34m[INFO][0m Setting up certificate permissions for Tempo...
[0;34m[INFO][0m Configuring certificate permissions for tempo (user: tempo)
[0;34m[INFO][0m Initializing certificate directory for obs-web-universe-main-dev...
[2026-02-05 03:42:00 UTC] USER=www-data EUID=0 PID=3563773 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-02-05 03:42:00 UTC] USER=www-data EUID=0 PID=3563782 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-02-05 03:42:00 UTC] USER=www-data EUID=0 PID=3563791 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-02-05 03:42:00 UTC] USER=www-data EUID=0 PID=3563800 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-web-universe-main-dev
[0;32m[OK][0m Certificate directory initialized: /etc/fastorder/observability/certs/obs-web-universe-main-dev
[0;34m[INFO][0m Setting file permissions...
[2026-02-05 03:42:00 UTC] USER=www-data EUID=0 PID=3563810 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/tempo-cert.pem
[2026-02-05 03:42:00 UTC] USER=www-data EUID=0 PID=3563819 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem
[2026-02-05 03:42:00 UTC] USER=www-data EUID=0 PID=3563828 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-web-universe-main-dev/tempo-key.pem
[0;34m[INFO][0m Setting file ownership...
[2026-02-05 03:42:00 UTC] USER=www-data EUID=0 PID=3563837 ACTION=passthru ARGS=chown root:tempo /etc/fastorder/observability/certs/obs-web-universe-main-dev/tempo-key.pem
[2026-02-05 03:42:00 UTC] USER=www-data EUID=0 PID=3563846 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-web-universe-main-dev/tempo-cert.pem /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem
[0;34m[INFO][0m Permission configuration completed
[0;34m[INFO][0m (Verification skipped - running via wrapper, trust chmod/chown success)
[0;32m[OK][0m ✅ Certificate permissions configured successfully for tempo
[2026-02-05 03:42:00 UTC] USER=www-data EUID=0 PID=3563865 ACTION=passthru ARGS=chown tempo:tempo /etc/tempo/obs-web-universe-main-dev/config.yaml
[2026-02-05 03:42:00 UTC] USER=www-data EUID=0 PID=3563874 ACTION=passthru ARGS=chmod 640 /etc/tempo/obs-web-universe-main-dev/config.yaml
[0;32m[OK][0m Configuration created at /etc/tempo/obs-web-universe-main-dev/config.yaml
[0;34m[INFO][0m Creating systemd service: tempo-obs-web-universe-main-dev
[0;32m[OK][0m Systemd service created
[0;34m[INFO][0m Adding /etc/hosts entry for traces-web-universe-main-dev-tempo.fastorder.com -> 10.100.1.239
[0;32m[OK][0m Updated /etc/hosts entry to use VM_IP
[0;34m[INFO][0m Storing Tempo configuration in AWS Secrets Manager (if aws CLI present)...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/web/universe/main/dev/tempo/server-a5xrXh",
"Name": "fastorder/observability/web/universe/main/dev/tempo/server",
"VersionId": "996180b1-fa5c-4e02-9e09-2b507466f1af"
}
[0;32m[OK][0m Tempo configuration stored/updated in AWS Secrets Manager: fastorder/observability/web/universe/main/dev/tempo/server
[1;33m[WARN][0m Port cleanup library not found, skipping automatic cleanup
[0;34m[INFO][0m Adding iptables redirect for Tempo internal communication (required for search)...
[0;34m[INFO][0m ╔════════════════════════════════════════════════════════════════════════╗
[0;34m[INFO][0m ║ TEMPO IPTABLES DNAT CONFIGURATION (Audit Log) ║
[0;34m[INFO][0m ╠════════════════════════════════════════════════════════════════════════╣
[0;34m[INFO][0m ║ OBS_CELL: obs-web-universe-main-dev
[0;34m[INFO][0m ║ VM_IP: 10.100.1.239
[0;34m[INFO][0m ║ GRPC_PORT: 9334 (unique: 9095 + last_octet)
[0;34m[INFO][0m ║ TEMPO_UID: 989
[0;34m[INFO][0m ║ TIMESTAMP: 2026-02-05T03:42:03Z
[0;34m[INFO][0m ╚════════════════════════════════════════════════════════════════════════╝
[0;34m[INFO][0m Using --uid-owner 989 for DNAT rule (scoped to tempo user)
[2026-02-05 03:42:03 UTC] USER=www-data EUID=0 PID=3563982 ACTION=passthru ARGS=iptables -t nat -A OUTPUT -p tcp -d 127.0.0.1 --dport 9334 -m owner --uid-owner 989 -j DNAT --to-destination 10.100.1.239:9334
ERROR: passthru not allowed: iptables
[0;31m[ERR][0m Could not add iptables redirect (iptables not allowed in wrapper)
[0;31m[ERR][0m ╔════════════════════════════════════════════════════════════════════════╗
[0;31m[ERR][0m ║ CRITICAL: Tempo search will NOT work without this redirect! ║
[0;31m[ERR][0m ║ ║
[0;31m[ERR][0m ║ Root cause: Tempo single-binary dials 127.0.0.1:<grpc_port> ║
[0;31m[ERR][0m ║ Each instance needs unique port + matching DNAT rule. ║
[0;31m[ERR][0m ║ ║
[0;31m[ERR][0m ║ Manually run: ║
[0;31m[ERR][0m ║ sudo iptables -t nat -A OUTPUT -p tcp -d 127.0.0.1 --dport 9334 -m owner --uid-owner 989 \ ║
[0;31m[ERR][0m ║ -j DNAT --to-destination 10.100.1.239:9334 ║
[0;31m[ERR][0m ╚════════════════════════════════════════════════════════════════════════╝
[0;34m[INFO][0m Enabling and starting Grafana Tempo service...
[2026-02-05 03:42:03 UTC] USER=www-data EUID=0 PID=3563990 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 03:42:03 UTC] USER=www-data EUID=0 PID=3564049 ACTION=passthru ARGS=systemctl enable tempo-obs-web-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/tempo-obs-web-universe-main-dev.service → /etc/systemd/system/tempo-obs-web-universe-main-dev.service.
[2026-02-05 03:42:04 UTC] USER=www-data EUID=0 PID=3564111 ACTION=passthru ARGS=systemctl restart tempo-obs-web-universe-main-dev.service
[0;32m[OK][0m Service enabled and started
[0;34m[INFO][0m Validating deployment...
[2026-02-05 03:42:07 UTC] USER=www-data EUID=0 PID=3564232 ACTION=passthru ARGS=systemctl is-active --quiet tempo-obs-web-universe-main-dev.service
[0;32m[OK][0m ✅ Grafana Tempo is running
[0;32m[OK][0m ✅ HTTP endpoint listening on port 3200
[0;32m[OK][0m ✅ OTLP gRPC endpoint listening on port 4317
[0;32m[OK][0m ✅ OTLP HTTP endpoint listening on port 4318
[0;34m[INFO][0m Running smoke test: Tempo search endpoint...
[1;33m[WARN][0m ⚠️ Tempo search smoke test failed - check iptables DNAT rule
[1;33m[WARN][0m Expected JSON with completedJobs/totalJobs, got: <html>
<head>
<title>Page Not Found</title>
<style>
body{
[0;34m[INFO][0m Service logs (last 10 lines):
[2026-02-05 03:42:09 UTC] USER=www-data EUID=0 PID=3564298 ACTION=passthru ARGS=journalctl -u tempo-obs-web-universe-main-dev.service -n 10 --no-pager
Feb 05 03:42:04 web-03 tempo-obs-web-universe-main-dev[3564143]: level=info ts=2026-02-05T03:42:04.982195252Z caller=compactor.go:125 msg="waiting until compactor is ACTIVE in the ring"
Feb 05 03:42:04 web-03 tempo-obs-web-universe-main-dev[3564143]: level=info ts=2026-02-05T03:42:04.981064744Z caller=worker.go:250 msg="total worker concurrency updated" totalConcurrency=20
Feb 05 03:42:04 web-03 tempo-obs-web-universe-main-dev[3564143]: level=info ts=2026-02-05T03:42:04.982809454Z caller=compactor.go:131 msg="compactor is ACTIVE in the ring"
Feb 05 03:42:04 web-03 tempo-obs-web-universe-main-dev[3564143]: level=info ts=2026-02-05T03:42:04.983106421Z caller=compactor.go:140 msg="waiting until compactor ring topology is stable" min_waiting=1m0s max_waiting=5m0s
Feb 05 03:42:04 web-03 tempo-obs-web-universe-main-dev[3564143]: level=warn ts=2026-02-05T03:42:04.983329197Z caller=wal.go:103 msg="unowned file entry ignored during wal replay" file=blocks err=null
Feb 05 03:42:04 web-03 tempo-obs-web-universe-main-dev[3564143]: level=info ts=2026-02-05T03:42:04.983508299Z caller=ingester.go:451 msg="wal replay complete"
Feb 05 03:42:04 web-03 tempo-obs-web-universe-main-dev[3564143]: level=info ts=2026-02-05T03:42:04.983758546Z caller=ingester.go:465 msg="reloading local blocks" tenants=0
Feb 05 03:42:04 web-03 tempo-obs-web-universe-main-dev[3564143]: level=info ts=2026-02-05T03:42:04.983971924Z caller=lifecycler.go:687 msg="not loading tokens from file, tokens file path is empty"
Feb 05 03:42:04 web-03 tempo-obs-web-universe-main-dev[3564143]: level=info ts=2026-02-05T03:42:04.984217313Z caller=lifecycler.go:714 msg="instance not found in ring, adding with no tokens" ring=ingester
Feb 05 03:42:04 web-03 tempo-obs-web-universe-main-dev[3564143]: level=info ts=2026-02-05T03:42:04.984475896Z caller=lifecycler.go:556 msg="auto-joining cluster after timeout" ring=ingester
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Traces Deployed Successfully
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: tempo
[0;34m[INFO][0m FQDN: traces-web-universe-main-dev-tempo.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.239
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Registering Tempo in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Tempo
[INFO] Identifier: web-universe-main-dev-tempo
[INFO] Identifier Parent: cluster
[INFO] IP: 10.100.1.239
[INFO] Port: 3200
[INFO] FQDN: traces-web-universe-main-dev-tempo.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[WARN] Registration API call failed (HTTP 500), retrying (1/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[WARN] Registration API call failed (HTTP 500), retrying (2/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ REGISTRATION FAILED AFTER 3 ATTEMPTS
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] HTTP Status: 500
[ERROR] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[ERROR]
[ERROR] API endpoint: https://skeleton.dev.fastorder.com/api/obs/register
[ERROR]
[ERROR] Troubleshooting:
[ERROR] 1. Check if skeleton.dev.fastorder.com is accessible
[ERROR] 2. Verify web application is running
[ERROR] 3. Check web application logs: /var/www/html/skeleton.dev.fastorder.com/logs/
[ERROR] 4. Test API manually:
[ERROR] curl -k -X POST 'https://skeleton.dev.fastorder.com/api/obs/register' \
[ERROR] -H 'Content-Type: application/json' \
[ERROR] -H 'X-Internal-Token: $OBS_INTERNAL_API_TOKEN' \
[ERROR] -d '$PAYLOAD'
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[1;33m[WARN][0m ⚠️ Failed to register Tempo (service is running)
[0;32m[OK][0m Traces backend (tempo) deployed successfully
[0;34m[INFO][0m Step 9/10: Deploying dashboards...
[0;34m[INFO][0m Provider: grafana (selected)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m DASHBOARDS DEPLOYMENT
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: grafana
[0;34m[INFO][0m Observability Cell: obs-web-universe-main-dev
[0;34m[INFO][0m FQDN: dashboards-web-universe-main-dev-grafana.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.237
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Using provider: grafana
[0;34m[INFO][0m Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Dashboards/provider/grafana.sh
[0;34m[INFO][0m Executing provider deployment script...
[0;34m[INFO][0m Parsed: SERVICE=web, ZONE=universe, BRANCH=main, ENV=dev
[0;34m[INFO][0m Binding to allocated IP: 10.100.1.237
[0;34m[INFO][0m Deploying Grafana for observability cell: obs-web-universe-main-dev
[0;34m[INFO][0m FQDN: dashboards-web-universe-main-dev-grafana.fastorder.com
[0;34m[INFO][0m Allocated IP: 10.100.1.237
[0;34m[INFO][0m VM IP: 10.100.1.237
[0;34m[INFO][0m HTTP Port: 3000
[0;34m[INFO][0m Checking if Grafana is installed...
[0;32m[OK][0m Grafana already installed
[0;34m[INFO][0m Installing Grafana plugins...
[0;34m[INFO][0m Installing ClickHouse datasource plugin...
[1;33m[WARN][0m Failed to install ClickHouse plugin (may need internet access)
[0;34m[INFO][0m Validating TLS certificate and key...
[0;34m[INFO][0m Creating certificate symlinks...
[0;32m[OK][0m Certificate symlinks created
[0;34m[INFO][0m Setting certificate permissions...
[0;32m[OK][0m TLS cert/key found and permissions set
[0;34m[INFO][0m Creating configuration and data directories...
[2026-02-05 03:42:15 UTC] USER=www-data EUID=0 PID=3564536 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-web-universe-main-dev
[2026-02-05 03:42:15 UTC] USER=www-data EUID=0 PID=3564545 ACTION=passthru ARGS=mkdir -p /var/lib/grafana/obs-web-universe-main-dev
[2026-02-05 03:42:15 UTC] USER=www-data EUID=0 PID=3564554 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-web-universe-main-dev/provisioning/datasources
[2026-02-05 03:42:15 UTC] USER=www-data EUID=0 PID=3564563 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-web-universe-main-dev/provisioning/dashboards
[2026-02-05 03:42:15 UTC] USER=www-data EUID=0 PID=3564573 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-web-universe-main-dev/provisioning/notifiers
[0;34m[INFO][0m Creating Grafana configuration at /etc/grafana/obs-web-universe-main-dev/grafana.ini...
[0;32m[OK][0m Configuration created
[0;34m[INFO][0m Creating Prometheus datasource provisioning...
[0;32m[OK][0m Prometheus datasource provisioned
[0;34m[INFO][0m Creating Tempo datasource provisioning...
[0;32m[OK][0m Tempo datasource provisioned
[0;34m[INFO][0m Creating Loki datasource provisioning...
[0;32m[OK][0m Loki datasource provisioned
[0;34m[INFO][0m Creating ClickHouse datasource provisioning...
[0;32m[OK][0m Retrieved ClickHouse credentials from Secrets Manager
[0;32m[OK][0m ClickHouse datasource provisioned
[0;34m[INFO][0m Creating systemd service: grafana-obs-web-universe-main-dev
[0;32m[OK][0m Systemd service created
[2026-02-05 03:42:18 UTC] USER=www-data EUID=0 PID=3564738 ACTION=passthru ARGS=chown -R grafana:grafana /etc/grafana/obs-web-universe-main-dev
[2026-02-05 03:42:18 UTC] USER=www-data EUID=0 PID=3564748 ACTION=passthru ARGS=chown -R grafana:grafana /var/lib/grafana/obs-web-universe-main-dev
[2026-02-05 03:42:18 UTC] USER=www-data EUID=0 PID=3564757 ACTION=passthru ARGS=chmod 750 /etc/grafana/obs-web-universe-main-dev /var/lib/grafana/obs-web-universe-main-dev
[0;34m[INFO][0m Adding /etc/hosts entry for dashboards-web-universe-main-dev-grafana.fastorder.com -> 10.100.1.237
[1;33m[WARN][0m /etc/hosts entry already exists
[0;34m[INFO][0m Storing Grafana credentials in AWS Secrets Manager (if aws CLI present)...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/web/universe/main/dev/grafana/admin-3TSJrX",
"Name": "fastorder/observability/web/universe/main/dev/grafana/admin",
"VersionId": "db6ff3cb-367f-4d7b-9b30-8b473d6e02ca"
}
[0;32m[OK][0m Credentials stored in AWS Secrets Manager: fastorder/observability/web/universe/main/dev/grafana/admin
[0;34m[INFO][0m Enabling and starting Grafana service...
[2026-02-05 03:42:20 UTC] USER=www-data EUID=0 PID=3564825 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 03:42:20 UTC] USER=www-data EUID=0 PID=3564886 ACTION=passthru ARGS=systemctl enable grafana-obs-web-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/grafana-obs-web-universe-main-dev.service → /etc/systemd/system/grafana-obs-web-universe-main-dev.service.
[2026-02-05 03:42:21 UTC] USER=www-data EUID=0 PID=3564941 ACTION=passthru ARGS=systemctl restart grafana-obs-web-universe-main-dev.service
[0;32m[OK][0m Service enabled and started
[0;34m[INFO][0m Validating deployment...
[2026-02-05 03:42:26 UTC] USER=www-data EUID=0 PID=3565096 ACTION=passthru ARGS=systemctl is-active --quiet grafana-obs-web-universe-main-dev.service
[0;32m[OK][0m ✅ Grafana is running
[0;32m[OK][0m ✅ Grafana web interface listening on port 3000
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m Grafana Dashboard URL: https://dashboards-web-universe-main-dev-grafana.fastorder.com:3000
[0;32m[OK][0m Username: admin
[0;32m[OK][0m Password is stored in AWS Secrets Manager at: fastorder/observability/web/universe/main/dev/grafana/admin
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Service logs (last 10 lines):
[2026-02-05 03:42:26 UTC] USER=www-data EUID=0 PID=3565110 ACTION=passthru ARGS=journalctl -u grafana-obs-web-universe-main-dev.service -n 10 --no-pager
Feb 05 03:42:26 web-03 grafana-obs-web-universe-main-dev[3564951]: logger=migrator t=2026-02-05T03:42:26.526265101Z level=info msg="Migration successfully executed" id="add snapshot local_directory column" duration=13.202243ms
Feb 05 03:42:26 web-03 grafana-obs-web-universe-main-dev[3564951]: logger=migrator t=2026-02-05T03:42:26.530357633Z level=info msg="Executing migration" id="add snapshot gms_snapshot_uid column"
Feb 05 03:42:26 web-03 grafana-obs-web-universe-main-dev[3564951]: logger=migrator t=2026-02-05T03:42:26.540925207Z level=info msg="Migration successfully executed" id="add snapshot gms_snapshot_uid column" duration=10.56552ms
Feb 05 03:42:26 web-03 grafana-obs-web-universe-main-dev[3564951]: logger=migrator t=2026-02-05T03:42:26.550296467Z level=info msg="Executing migration" id="add snapshot encryption_key column"
Feb 05 03:42:26 web-03 grafana-obs-web-universe-main-dev[3564951]: logger=migrator t=2026-02-05T03:42:26.565502055Z level=info msg="Migration successfully executed" id="add snapshot encryption_key column" duration=15.202432ms
Feb 05 03:42:26 web-03 grafana-obs-web-universe-main-dev[3564951]: logger=migrator t=2026-02-05T03:42:26.570027133Z level=info msg="Executing migration" id="add snapshot error_string column"
Feb 05 03:42:26 web-03 grafana-obs-web-universe-main-dev[3564951]: logger=migrator t=2026-02-05T03:42:26.580846488Z level=info msg="Migration successfully executed" id="add snapshot error_string column" duration=10.815398ms
Feb 05 03:42:26 web-03 grafana-obs-web-universe-main-dev[3564951]: logger=migrator t=2026-02-05T03:42:26.586156706Z level=info msg="Executing migration" id="create cloud_migration_resource table v1"
Feb 05 03:42:26 web-03 grafana-obs-web-universe-main-dev[3564951]: logger=migrator t=2026-02-05T03:42:26.588852622Z level=info msg="Migration successfully executed" id="create cloud_migration_resource table v1" duration=2.69762ms
Feb 05 03:42:26 web-03 grafana-obs-web-universe-main-dev[3564951]: logger=migrator t=2026-02-05T03:42:26.593259435Z level=info msg="Executing migration" id="delete cloud_migration_snapshot.result column"
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Dashboards Deployed Successfully
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: grafana
[0;34m[INFO][0m FQDN: dashboards-web-universe-main-dev-grafana.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.237
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Registering Grafana in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Grafana
[INFO] Identifier: web-universe-main-dev-grafana
[INFO] Identifier Parent: cluster
[INFO] IP: 10.100.1.237
[INFO] Port: 3000
[INFO] FQDN: dashboards-web-universe-main-dev-grafana.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[WARN] Registration API call failed (HTTP 500), retrying (1/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[WARN] Registration API call failed (HTTP 500), retrying (2/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ REGISTRATION FAILED AFTER 3 ATTEMPTS
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] HTTP Status: 500
[ERROR] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[ERROR]
[ERROR] API endpoint: https://skeleton.dev.fastorder.com/api/obs/register
[ERROR]
[ERROR] Troubleshooting:
[ERROR] 1. Check if skeleton.dev.fastorder.com is accessible
[ERROR] 2. Verify web application is running
[ERROR] 3. Check web application logs: /var/www/html/skeleton.dev.fastorder.com/logs/
[ERROR] 4. Test API manually:
[ERROR] curl -k -X POST 'https://skeleton.dev.fastorder.com/api/obs/register' \
[ERROR] -H 'Content-Type: application/json' \
[ERROR] -H 'X-Internal-Token: $OBS_INTERNAL_API_TOKEN' \
[ERROR] -d '$PAYLOAD'
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[1;33m[WARN][0m ⚠️ Failed to register Grafana (service is running)
[0;32m[OK][0m Dashboards (grafana) deployed successfully
[0;34m[INFO][0m Step 10/10: Deploying alerting...
[0;34m[INFO][0m Provider: alertmanager (selected)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m ALERTING DEPLOYMENT
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: alertmanager
[0;34m[INFO][0m Observability Cell: obs-web-universe-main-dev
[0;34m[INFO][0m FQDN: alerts-web-universe-main-dev-alertmanager.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.240
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Ports: Web=9093 Cluster=9094 (bound to IP: 10.100.1.240)
[0;34m[INFO][0m Using provider: alertmanager
[0;34m[INFO][0m Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Alerting/provider/alertmanager.sh
[0;34m[INFO][0m Executing provider deployment script...
[0;34m[INFO][0m Parsed: SERVICE=web, ZONE=universe, BRANCH=main, ENV=dev
[0;34m[INFO][0m Binding to allocated IP: 10.100.1.240
[0;34m[INFO][0m Deploying Alertmanager for observability cell: obs-web-universe-main-dev
[0;34m[INFO][0m FQDN: alerts-web-universe-main-dev-alertmanager.fastorder.com
[0;34m[INFO][0m Allocated IP: 10.100.1.240
[0;34m[INFO][0m VM IP: 10.100.1.240
[0;34m[INFO][0m Ports: Web=9093 Cluster=9094
[0;34m[VERSION][0m Fetching latest version for alertmanager from GitHub (prometheus/alertmanager)...
[0;32m[VERSION][0m Latest alertmanager version: 0.31.0
[0;34m[INFO][0m Resolved Alertmanager version: 0.31.0
[0;34m[INFO][0m Checking if Alertmanager is installed...
[0;32m[OK][0m Alertmanager already installed at /usr/local/bin/alertmanager
[0;34m[INFO][0m Validating TLS certificate and key...
[0;32m[OK][0m TLS cert/key found in /etc/fastorder/observability/certs/obs-web-universe-main-dev
[0;34m[INFO][0m Creating configuration and data directories...
[2026-02-05 03:42:32 UTC] USER=www-data EUID=0 PID=3565372 ACTION=passthru ARGS=mkdir -p /etc/alertmanager/obs-web-universe-main-dev
[2026-02-05 03:42:32 UTC] USER=www-data EUID=0 PID=3565381 ACTION=passthru ARGS=mkdir -p /var/lib/alertmanager/obs-web-universe-main-dev
[2026-02-05 03:42:32 UTC] USER=www-data EUID=0 PID=3565390 ACTION=passthru ARGS=mkdir -p /etc/alertmanager/obs-web-universe-main-dev/templates
[0;34m[INFO][0m Creating Alertmanager configuration...
[0;32m[OK][0m Alertmanager configuration created at /etc/alertmanager/obs-web-universe-main-dev/alertmanager.yml
[0;34m[INFO][0m Creating notification templates...
[0;32m[OK][0m Notification templates created
[0;34m[INFO][0m Creating Alertmanager web TLS configuration with mTLS...
[0;32m[OK][0m Web mTLS configuration created at /etc/alertmanager/obs-web-universe-main-dev/web-config.yml
[0;34m[INFO][0m Validating Alertmanager configuration...
[2026-02-05 03:42:32 UTC] USER=www-data EUID=0 PID=3565426 ACTION=passthru ARGS=chmod 755 /etc/alertmanager/obs-web-universe-main-dev
[2026-02-05 03:42:32 UTC] USER=www-data EUID=0 PID=3565435 ACTION=passthru ARGS=chmod 644 /etc/alertmanager/obs-web-universe-main-dev/alertmanager.yml
Checking '/etc/alertmanager/obs-web-universe-main-dev/alertmanager.yml' SUCCESS
Found:
- global config
- route
- 6 inhibit rules
- 5 receivers
- 1 templates
SUCCESS
[0;32m[OK][0m ✅ Configuration is valid
[0;34m[INFO][0m Creating systemd service: alertmanager-obs-web-universe-main-dev
[0;32m[OK][0m Systemd service created
[2026-02-05 03:42:33 UTC] USER=www-data EUID=0 PID=3565461 ACTION=passthru ARGS=chown alertmanager:alertmanager /etc/fastorder/observability/certs/obs-web-universe-main-dev/alertmanager-key.pem
[2026-02-05 03:42:33 UTC] USER=www-data EUID=0 PID=3565470 ACTION=passthru ARGS=chown alertmanager:alertmanager /etc/fastorder/observability/certs/obs-web-universe-main-dev/alertmanager-cert.pem
[2026-02-05 03:42:33 UTC] USER=www-data EUID=0 PID=3565479 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-web-universe-main-dev/ca-cert.pem
[2026-02-05 03:42:33 UTC] USER=www-data EUID=0 PID=3565488 ACTION=passthru ARGS=chown -R alertmanager:alertmanager /etc/alertmanager/obs-web-universe-main-dev
[2026-02-05 03:42:33 UTC] USER=www-data EUID=0 PID=3565497 ACTION=passthru ARGS=chown -R alertmanager:alertmanager /var/lib/alertmanager/obs-web-universe-main-dev
[2026-02-05 03:42:33 UTC] USER=www-data EUID=0 PID=3565507 ACTION=passthru ARGS=chmod 750 /etc/alertmanager/obs-web-universe-main-dev /var/lib/alertmanager/obs-web-universe-main-dev
[0;34m[INFO][0m Adding /etc/hosts entry for alerts-web-universe-main-dev-alertmanager.fastorder.com -> 10.100.1.240
[1;33m[WARN][0m /etc/hosts entry already exists
[0;34m[INFO][0m Storing Alertmanager configuration in AWS Secrets Manager (if aws CLI present)...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/web/universe/main/dev/alertmanager/server-Q7G62t",
"Name": "fastorder/observability/web/universe/main/dev/alertmanager/server",
"VersionId": "8c98947b-096a-43e7-b972-a509eda01455"
}
[0;32m[OK][0m Configuration stored in AWS Secrets Manager: fastorder/observability/web/universe/main/dev/alertmanager/server
[0;34m[INFO][0m Enabling and starting Alertmanager service...
[2026-02-05 03:42:34 UTC] USER=www-data EUID=0 PID=3565560 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 03:42:35 UTC] USER=www-data EUID=0 PID=3565672 ACTION=passthru ARGS=systemctl enable alertmanager-obs-web-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/alertmanager-obs-web-universe-main-dev.service → /etc/systemd/system/alertmanager-obs-web-universe-main-dev.service.
[2026-02-05 03:42:35 UTC] USER=www-data EUID=0 PID=3565726 ACTION=passthru ARGS=systemctl restart alertmanager-obs-web-universe-main-dev.service
[0;32m[OK][0m Service enabled and started
[0;34m[INFO][0m Validating deployment...
[0;32m[OK][0m ✅ Alertmanager is running
[0;32m[OK][0m ✅ Alertmanager HTTPS web interface listening on port 9093
[0;32m[OK][0m ✅ Alertmanager cluster port listening on port 9094
[1;33m[WARN][0m ⚠️ Alertmanager health check not responding yet (HTTPS)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m Alertmanager Web UI: https://alerts-web-universe-main-dev-alertmanager.fastorder.com:9093
[0;32m[OK][0m API Endpoint: https://alerts-web-universe-main-dev-alertmanager.fastorder.com:9093/api/v2
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Service logs (last 10 lines):
[2026-02-05 03:42:38 UTC] USER=www-data EUID=0 PID=3565915 ACTION=passthru ARGS=journalctl -u alertmanager-obs-web-universe-main-dev.service -n 10 --no-pager
Feb 05 03:42:35 web-03 systemd[1]: Started Alertmanager - obs-web-universe-main-dev.
Feb 05 03:42:35 web-03 alertmanager-obs-web-universe-main-dev[3565745]: ts=2026-02-05T03:42:35.769Z caller=main.go:245 level=info msg="Starting Alertmanager" version="(version=0.26.0, branch=HEAD, revision=d7b4f0c7322e7151d6e3b1e31cbc15361e295d8d)"
Feb 05 03:42:35 web-03 alertmanager-obs-web-universe-main-dev[3565745]: ts=2026-02-05T03:42:35.770Z caller=main.go:246 level=info build_context="(go=go1.20.7, platform=linux/amd64, user=root@df8d7debeef4, date=20230824-11:11:58, tags=netgo)"
Feb 05 03:42:35 web-03 alertmanager-obs-web-universe-main-dev[3565745]: ts=2026-02-05T03:42:35.775Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s
Feb 05 03:42:35 web-03 alertmanager-obs-web-universe-main-dev[3565745]: ts=2026-02-05T03:42:35.847Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/obs-web-universe-main-dev/alertmanager.yml
Feb 05 03:42:35 web-03 alertmanager-obs-web-universe-main-dev[3565745]: ts=2026-02-05T03:42:35.850Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/obs-web-universe-main-dev/alertmanager.yml
Feb 05 03:42:35 web-03 alertmanager-obs-web-universe-main-dev[3565745]: ts=2026-02-05T03:42:35.854Z caller=tls_config.go:274 level=info msg="Listening on" address=10.100.1.240:9093
Feb 05 03:42:35 web-03 alertmanager-obs-web-universe-main-dev[3565745]: ts=2026-02-05T03:42:35.856Z caller=tls_config.go:310 level=info msg="TLS is enabled." http2=true address=10.100.1.240:9093
Feb 05 03:42:37 web-03 alertmanager-obs-web-universe-main-dev[3565745]: ts=2026-02-05T03:42:37.776Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.0011014s
Feb 05 03:42:38 web-03 alertmanager-obs-web-universe-main-dev[3565745]: 2026/02/05 03:42:38 http: TLS handshake error from 10.100.1.240:59614: tls: client didn't provide a certificate
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Alerting Deployed Successfully
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: alertmanager
[0;34m[INFO][0m FQDN: alerts-web-universe-main-dev-alertmanager.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.240
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Setting up HTTPS reverse proxy...
[0;34m[INFO][0m Backend port: 9093
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Alertmanager HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OBS Cell: obs-web-universe-main-dev
FQDN: alerts-web-universe-main-dev-alertmanager.fastorder.com
Backend: https://alerts-web-universe-main-dev-alertmanager.fastorder.com:9093/ (resolved via /etc/hosts)
Backend IP: 10.100.1.240
Email: admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verifying prerequisites...
[0;31m[ERROR][0m This script must be run as root or with sudo
[1;33m[WARN][0m ⚠️ HTTPS setup failed (Alertmanager is still running on HTTP)
[0;34m[INFO][0m Registering Alertmanager in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Alertmanager
[INFO] Identifier: web-universe-main-dev-alertmanager
[INFO] Identifier Parent: cluster
[INFO] IP: 10.100.1.240
[INFO] Port: 9093
[INFO] FQDN: alerts-web-universe-main-dev-alertmanager.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[WARN] Registration API call failed (HTTP 500), retrying (1/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[WARN] Registration API call failed (HTTP 500), retrying (2/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ REGISTRATION FAILED AFTER 3 ATTEMPTS
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] HTTP Status: 500
[ERROR] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[ERROR]
[ERROR] API endpoint: https://skeleton.dev.fastorder.com/api/obs/register
[ERROR]
[ERROR] Troubleshooting:
[ERROR] 1. Check if skeleton.dev.fastorder.com is accessible
[ERROR] 2. Verify web application is running
[ERROR] 3. Check web application logs: /var/www/html/skeleton.dev.fastorder.com/logs/
[ERROR] 4. Test API manually:
[ERROR] curl -k -X POST 'https://skeleton.dev.fastorder.com/api/obs/register' \
[ERROR] -H 'Content-Type: application/json' \
[ERROR] -H 'X-Internal-Token: $OBS_INTERNAL_API_TOKEN' \
[ERROR] -d '$PAYLOAD'
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[1;33m[WARN][0m ⚠️ Failed to register Alertmanager (service is running)
[0;32m[OK][0m Alerting (alertmanager) deployed successfully
[0;34m[INFO][0m Step 10.5: Deploying Blackbox Exporter for synthetic monitoring...
[0;34m[VERSION][0m Fetching latest version for blackbox_exporter from GitHub (prometheus/blackbox_exporter)...
[0;32m[VERSION][0m Latest blackbox_exporter version: 0.28.0
[0;32m[BLACKBOX][0m Resolved Blackbox Exporter version: 0.28.0
[0;32m[BLACKBOX][0m Starting Blackbox Exporter deployment for obs-web-universe-main-dev
[0;32m[BLACKBOX][0m VM IP: 10.100.1.236
[0;32m[BLACKBOX][0m Version: 0.28.0
[0;32m[BLACKBOX][0m Checking prerequisites...
[0;32m[BLACKBOX][0m Creating directories...
[0;32m[BLACKBOX][0m Downloading Blackbox Exporter v0.28.0...
Sorry, user www-data is not allowed to execute '/usr/bin/mv /tmp/tmp.u7canr1Cky/blackbox_exporter-0.28.0.linux-amd64/blackbox_exporter /usr/local/bin/' as root on web-03.
[1;33m[WARN][0m Blackbox Exporter deployment failed (non-fatal, synthetic monitoring disabled)
[0;34m[INFO][0m Step 11/13: Configuring HTTPS reverse proxies...
[0;34m[INFO][0m Setting up Prometheus HTTPS proxy...
[2026-02-05 03:42:46 UTC] USER=www-data EUID=0 PID=3566291 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/https/setup-prometheus-https.sh --obs-cell obs-web-universe-main-dev --backend-ip 10.100.1.236
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Prometheus HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OBS Cell: obs-web-universe-main-dev
FQDN: metrics-web-universe-main-dev-prometheus.fastorder.com
Backend: https://metrics-web-universe-main-dev-prometheus.fastorder.com:9090/ (resolved via /etc/hosts)
Backend IP: 10.100.1.236
Email: admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verifying prerequisites...
[0;34m[INFO][0m Enabling Apache modules...
[0;34m[INFO][0m Testing backend connectivity (will retry up to 60s)...
[0;32m[OK][0m Backend is accessible
[0;34m[INFO][0m Creating HTTP VirtualHost for ACME challenge...
[0;32m[OK][0m HTTP VirtualHost created
[0;34m[INFO][0m Obtaining Let's Encrypt certificate...
Another instance of Certbot is already running.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/tmppe8_osss/log or re-run Certbot with -v for more details.
[1;33m[WARN][0m Prometheus HTTPS proxy setup failed (non-fatal)
[0;34m[INFO][0m Setting up Grafana HTTPS proxy...
[2026-02-05 03:42:47 UTC] USER=www-data EUID=0 PID=3566364 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Dashboards/https/setup-grafana-https.sh --obs-cell obs-web-universe-main-dev --backend-ip 10.100.1.237
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OBS Cell: obs-web-universe-main-dev
FQDN: dashboards-web-universe-main-dev-grafana.fastorder.com
Backend: https://dashboards-web-universe-main-dev-grafana.fastorder.com:3000/ (resolved via /etc/hosts)
Backend IP: 10.100.1.237
Email: admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verifying prerequisites...
[0;34m[INFO][0m Enabling Apache modules...
[0;34m[INFO][0m Testing backend connectivity...
[0;34m[INFO][0m Creating HTTP VirtualHost for ACME challenge...
[0;32m[OK][0m HTTP VirtualHost created
[0;34m[INFO][0m Obtaining Let's Encrypt certificate...
Another instance of Certbot is already running.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/tmpqy9j4g4c/log or re-run Certbot with -v for more details.
[1;33m[WARN][0m Grafana HTTPS proxy setup failed (non-fatal)
[0;34m[INFO][0m Setting up OpenTelemetry Collector HTTPS proxy...
[2026-02-05 03:42:48 UTC] USER=www-data EUID=0 PID=3566460 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Telemetry/https/setup-otelcol-https.sh --obs-cell obs-web-universe-main-dev --backend-ip 10.100.1.241
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OpenTelemetry Collector HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OBS Cell: obs-web-universe-main-dev
FQDN: telemetry-web-universe-main-dev-opentelemetry.fastorder.com
Backend: http://telemetry-web-universe-main-dev-opentelemetry.fastorder.com:8888/ (resolved via /etc/hosts)
Backend IP: 10.100.1.241
Email: admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verifying prerequisites...
[0;34m[INFO][0m Enabling Apache modules...
[0;34m[INFO][0m Testing backend connectivity...
[1;33m[WARN][0m Cannot verify OTel Collector metrics endpoint, continuing anyway...
[0;34m[INFO][0m Creating HTTP VirtualHost for ACME challenge...
[0;32m[OK][0m HTTP VirtualHost created
[0;34m[INFO][0m Obtaining Let's Encrypt certificate...
Another instance of Certbot is already running.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/tmpbuuvw39b/log or re-run Certbot with -v for more details.
[1;33m[WARN][0m OpenTelemetry Collector HTTPS proxy setup failed (non-fatal)
[0;34m[INFO][0m Setting up ClickHouse HTTPS proxy...
[2026-02-05 03:42:49 UTC] USER=www-data EUID=0 PID=3566531 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/https/setup-clickhouse-https.sh --obs-cell obs-web-universe-main-dev --backend-ip 10.100.1.238
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ClickHouse HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OBS Cell: obs-web-universe-main-dev
FQDN: logstore-web-universe-main-dev.fastorder.com
Backend: http://logstore-web-universe-main-dev.fastorder.com:8123/ (resolved via /etc/hosts)
Backend IP: 10.100.1.238
Email: admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verifying prerequisites...
[0;34m[INFO][0m Enabling Apache modules...
[0;34m[INFO][0m Testing backend connectivity (will retry up to 60s)...
[0;32m[OK][0m Backend is accessible
[0;34m[INFO][0m Creating HTTP VirtualHost for ACME challenge...
[0;32m[OK][0m HTTP VirtualHost created
[0;34m[INFO][0m Obtaining Let's Encrypt certificate...
Another instance of Certbot is already running.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/tmpih9i4jz_/log or re-run Certbot with -v for more details.
[1;33m[WARN][0m ClickHouse HTTPS proxy setup failed (non-fatal)
[0;34m[INFO][0m Setting up Tempo HTTPS proxy...
[2026-02-05 03:42:50 UTC] USER=www-data EUID=0 PID=3566601 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Traces/https/setup-tempo-https.sh --obs-cell obs-web-universe-main-dev --backend-ip 10.100.1.239
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana Tempo HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OBS Cell: obs-web-universe-main-dev
FQDN: traces-web-universe-main-dev-tempo.fastorder.com
Backend: https://10.100.1.239:3200/
Backend IP: 10.100.1.239
Email: admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verifying prerequisites...
[0;34m[INFO][0m Enabling Apache modules...
[0;34m[INFO][0m Testing backend connectivity...
[1;33m[WARN][0m Cannot verify Tempo health endpoint (it may not be running yet), continuing anyway...
[0;34m[INFO][0m Creating HTTP VirtualHost for ACME challenge...
[0;32m[OK][0m HTTP VirtualHost created
[0;34m[INFO][0m Obtaining Let's Encrypt certificate...
Another instance of Certbot is already running.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/tmp6umbrqmu/log or re-run Certbot with -v for more details.
[1;33m[WARN][0m Tempo HTTPS proxy setup failed (non-fatal)
[0;34m[INFO][0m Setting up Alertmanager HTTPS proxy...
[2026-02-05 03:42:51 UTC] USER=www-data EUID=0 PID=3566686 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Alerting/https/setup-alertmanager-https.sh --obs-cell obs-web-universe-main-dev --backend-ip 10.100.1.240
[0;34m[INFO][0m Backend port: 9093
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Alertmanager HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OBS Cell: obs-web-universe-main-dev
FQDN: alerts-web-universe-main-dev-alertmanager.fastorder.com
Backend: https://alerts-web-universe-main-dev-alertmanager.fastorder.com:9093/ (resolved via /etc/hosts)
Backend IP: 10.100.1.240
Email: admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verifying prerequisites...
[0;34m[INFO][0m Enabling Apache modules...
[0;34m[INFO][0m Testing backend connectivity...
[1;33m[WARN][0m Backend health check inconclusive - proceeding anyway
[0;34m[INFO][0m Creating HTTP VirtualHost for ACME challenge...
[0;32m[OK][0m HTTP VirtualHost created
[0;34m[INFO][0m Obtaining Let's Encrypt certificate...
Another instance of Certbot is already running.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/tmp_tkg9d6f/log or re-run Certbot with -v for more details.
[1;33m[WARN][0m Alertmanager HTTPS proxy setup failed (non-fatal)
[0;32m[OK][0m HTTPS reverse proxies configured
[0;34m[INFO][0m Step 12/13: Configuring firewall rules (network segmentation)...
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING FIREWALL RULES FOR OBSERVABILITY CELL
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Cell ID: obs-web-universe-main-dev
[0;34m[INFO][0m Internal Network: 10.0.0.0/8
[0;34m[INFO][0m Discovering dashboard/skeleton VM IPs...
[0;34m[INFO][0m Discovered skeleton IP: 142.93.238.16 (skeleton.fastorder.com)
[0;34m[INFO][0m Authorized dashboard IPs:
[0;34m[INFO][0m - 10.100.60.2
[0;34m[INFO][0m - 142.93.238.16
[0;34m[INFO][0m Configuring UFW firewall rules...
[2026-02-05 03:42:52 UTC] USER=www-data EUID=0 PID=3566769 ACTION=passthru ARGS=ufw --force enable
ERROR: passthru not allowed: ufw
[2026-02-05 03:42:52 UTC] USER=www-data EUID=0 PID=3566777 ACTION=passthru ARGS=ufw default deny incoming
ERROR: passthru not allowed: ufw
[2026-02-05 03:42:52 UTC] USER=www-data EUID=0 PID=3566785 ACTION=passthru ARGS=ufw default allow outgoing
ERROR: passthru not allowed: ufw
[2026-02-05 03:42:52 UTC] USER=www-data EUID=0 PID=3566794 ACTION=passthru ARGS=ufw allow 22/tcp comment SSH
ERROR: passthru not allowed: ufw
[0;34m[INFO][0m Allowing prometheus (port 9090) from internal network...
[0;34m[INFO][0m Allowing alertmanager (port 9093) from internal network...
[2026-02-05 03:42:52 UTC] USER=www-data EUID=0 PID=3566811 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 9093 proto tcp comment Obs: alertmanager from internal
ERROR: passthru not allowed: ufw
[0;34m[INFO][0m Allowing clickhouse (port 8123) from internal network...
[2026-02-05 03:42:52 UTC] USER=www-data EUID=0 PID=3566819 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 8123 proto tcp comment Obs: clickhouse from internal
ERROR: passthru not allowed: ufw
[0;34m[INFO][0m Allowing grafana (port 3000) from internal network...
[2026-02-05 03:42:52 UTC] USER=www-data EUID=0 PID=3566827 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 3000 proto tcp comment Obs: grafana from internal
ERROR: passthru not allowed: ufw
[0;34m[INFO][0m Allowing otelcol (port 4318) from internal network...
[0;34m[INFO][0m Allowing loki (port 3100) from internal network...
[0;34m[INFO][0m Allowing tempo (port 3200) from internal network...
[0;34m[INFO][0m Allowing dashboard access from 10.100.60.2...
[2026-02-05 03:42:52 UTC] USER=www-data EUID=0 PID=3566859 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 9090 proto tcp comment Dashboard: prometheus
ERROR: passthru not allowed: ufw
[2026-02-05 03:42:52 UTC] USER=www-data EUID=0 PID=3566875 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 8123 proto tcp comment Dashboard: clickhouse
ERROR: passthru not allowed: ufw
[2026-02-05 03:42:52 UTC] USER=www-data EUID=0 PID=3566883 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3000 proto tcp comment Dashboard: grafana
ERROR: passthru not allowed: ufw
[2026-02-05 03:42:52 UTC] USER=www-data EUID=0 PID=3566891 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 4318 proto tcp comment Dashboard: otelcol
ERROR: passthru not allowed: ufw
[2026-02-05 03:42:52 UTC] USER=www-data EUID=0 PID=3566899 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3100 proto tcp comment Dashboard: loki
ERROR: passthru not allowed: ufw
[2026-02-05 03:42:52 UTC] USER=www-data EUID=0 PID=3566907 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3200 proto tcp comment Dashboard: tempo
ERROR: passthru not allowed: ufw
[0;34m[INFO][0m Allowing dashboard access from 142.93.238.16...
[2026-02-05 03:42:53 UTC] USER=www-data EUID=0 PID=3566941 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 3000 proto tcp comment Dashboard: grafana
ERROR: passthru not allowed: ufw
[2026-02-05 03:42:53 UTC] USER=www-data EUID=0 PID=3566949 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 4318 proto tcp comment Dashboard: otelcol
ERROR: passthru not allowed: ufw
[0;32m[OK][0m UFW firewall rules configured
[0;32m[OK][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Firewall configuration completed
[0;32m[OK][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Current firewall status:
[2026-02-05 03:42:53 UTC] USER=www-data EUID=0 PID=3566991 ACTION=passthru ARGS=ufw status numbered
ERROR: passthru not allowed: ufw
[0;32m[OK][0m Firewall rules configured
[0;34m[INFO][0m Step 13/13: Configuring OAuth/SSO...
[0;34m[INFO][0m OAuth/SSO configuration script not found, skipping...
[0;34m[INFO][0m Running validation checks...
[0;34m[INFO][0m Validation script not found, skipping...
[0;34m[INFO][0m Registering observability components to dashboard...
[0;34m[INFO][0m Components to register: metrics alerts dashboards traces telemetry logstore proxy
[0;34m[INFO][0m Skipping metrics - registered by deploy script
[0;34m[INFO][0m Skipping alerts - registered by deploy script
[0;34m[INFO][0m Skipping dashboards - registered by deploy script
[0;34m[INFO][0m Skipping traces - registered by deploy script
[0;34m[INFO][0m Skipping telemetry - registered by deploy script
[0;34m[INFO][0m Skipping logstore - registered by deploy script
[0;34m[INFO][0m Processing component: proxy
[0;34m[INFO][0m Registering: proxy (obs-web-universe-main-dev-proxy)
[INFO] Detected observability component, parsing: web-universe-main-dev-proxy
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Observability Proxy
[INFO] Identifier: obs-web-universe-main-dev-proxy
[INFO] Identifier Parent: observability-cell
[INFO] IP: 10.100.1.235
[INFO] Port: 443
[INFO] FQDN: observe-web-universe-main-dev.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[WARN] Registration API call failed (HTTP 500), retrying (1/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[WARN] Registration API call failed (HTTP 500), retrying (2/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ REGISTRATION FAILED AFTER 3 ATTEMPTS
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] HTTP Status: 500
[ERROR] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[ERROR]
[ERROR] API endpoint: https://skeleton.dev.fastorder.com/api/obs/register
[ERROR]
[ERROR] Troubleshooting:
[ERROR] 1. Check if skeleton.dev.fastorder.com is accessible
[ERROR] 2. Verify web application is running
[ERROR] 3. Check web application logs: /var/www/html/skeleton.dev.fastorder.com/logs/
[ERROR] 4. Test API manually:
[ERROR] curl -k -X POST 'https://skeleton.dev.fastorder.com/api/obs/register' \
[ERROR] -H 'Content-Type: application/json' \
[ERROR] -H 'X-Internal-Token: $OBS_INTERNAL_API_TOKEN' \
[ERROR] -d '$PAYLOAD'
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[1;33m[WARN][0m ⚠️ Registration returned non-zero for proxy (non-fatal)
[0;34m[INFO][0m Registering short DNS aliases...
[1;33m[WARN][0m ⚠️ No observability components were registered
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying all observability services are running...
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m[OK][0m ✓ prometheus-obs-web-universe-main-dev.service is running
[0;32m[OK][0m ✓ tempo-obs-web-universe-main-dev.service is running
[0;32m[OK][0m ✓ otelcol-obs-web-universe-main-dev.service is running
[0;32m[OK][0m ✓ All observability services verified running
═══════════════════════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ OBSERVABILITY CELL PROVISIONED: obs-web-universe-main-dev
═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m DNS Entries:
metrics-web-universe-main-dev-prometheus.fastorder.com (10.100.1.236)
alerts-web-universe-main-dev-alertmanager.fastorder.com (10.100.1.240)
dashboards-web-universe-main-dev-grafana.fastorder.com (10.100.1.237)
traces-web-universe-main-dev-tempo.fastorder.com (10.100.1.239)
telemetry-web-universe-main-dev-opentelemetry.fastorder.com (10.100.1.241)
logstore-web-universe-main-dev-clickhouse.fastorder.com (10.100.1.238)
observe-web-universe-main-dev.fastorder.com (10.100.1.235)
[0;34m[INFO][0m Secrets Path: fastorder/observability/web/universe/dev/*
[0;34m[INFO][0m Access (Purpose-Oriented URLs):
Dashboards: https://dashboards-web-universe-main-dev-grafana.fastorder.com (SSO enabled)
Metrics: https://metrics-web-universe-main-dev-prometheus.fastorder.com (internal only)
Alerts: https://alerts-web-universe-main-dev-alertmanager.fastorder.com
Log Storage: https://logstore-web-universe-main-dev-clickhouse.fastorder.com
[0;34m[INFO][0m Backend Implementation (Internal - Not Exposed to Clients):
Telemetry: otlp
Metrics: prometheus
Traces: tempo
Dashboards: grafana
Alerting: alertmanager
Log Storage: clickhouse
[0;34m[INFO][0m For applications in web-universe-main-dev:
- Metrics: Push to telemetry-web-universe-main-dev-opentelemetry.fastorder.com:4318 (OTLP/HTTP)
- Logs: Push to telemetry-web-universe-main-dev-opentelemetry.fastorder.com:4318 (OTLP/HTTP)
- Traces: Push to telemetry-web-universe-main-dev-opentelemetry.fastorder.com:4317 (OTLP/gRPC)
- Query Metrics: https://metrics-web-universe-main-dev-prometheus.fastorder.com
- Query Logs: https://logstore-web-universe-main-dev-clickhouse.fastorder.com
- Query Traces: https://traces-web-universe-main-dev-tempo.fastorder.com
[0;34m[INFO][0m Runbook: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/RUNBOOK.md
═══════════════════════════════════════════════════════════════════════════════
⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...[0;34m[INFO][0m Using eventbus engine from EVENTBUS_ENGINE environment variable: kafka
[0;34m[INFO][0m Cleaning up any existing locks...
[0;32m[1mStarting eventbus engine: kafka[0m
[1;33m═══════════════════════════════════════════════[0m
[0;34m[INFO][0m Using environment from web interface: web-universe-main-dev
[0;32m[2026-02-05 03:43:13][0m Using web-provided environment: web-universe-main-dev
[0;32m[2026-02-05 03:43:13][0m Service: web, Zone: universe, Branch: main, Env: dev
[0;32m✓[0m Environment initialized successfully (mode: general)
[0;34m[INFO][0m Starting Kafka setup process...
[0;34m[INFO][0m Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Found 13 step(s) to execute
[0;34m[INFO][0m 📦 Step 1/13: install debezium connector...
═══════════════════════════════════════════════════════════════════
Fetching latest versions from Maven Central...
Installing Debezium PostgreSQL Connector
Debezium version: 3.4.1.Final
pgjdbc version: 42.7.9
═══════════════════════════════════════════════════════════════════
[INFO] Downloading Debezium PostgreSQL connector 3.4.1.Final...
[INFO] Downloading PostgreSQL JDBC driver 42.7.9...
[INFO] Backing up existing installation to /usr/share/confluent-hub-components/debezium-connector-postgres.backup.20260205034314
[2026-02-05 03:43:14 UTC] USER=www-data EUID=0 PID=3567602 ACTION=fsop ARGS=mv /usr/share/confluent-hub-components/debezium-connector-postgres /usr/share/confluent-hub-components/debezium-connector-postgres.backup.20260205034314
[INFO] Extracting Debezium connector...
[INFO] Replacing pgjdbc with version 42.7.9...
[INFO] Installing to /usr/share/confluent-hub-components/debezium-connector-postgres...
[2026-02-05 03:43:14 UTC] USER=www-data EUID=0 PID=3567618 ACTION=fsop ARGS=mkdir -p /usr/share/confluent-hub-components
[2026-02-05 03:43:14 UTC] USER=www-data EUID=0 PID=3567630 ACTION=fsop ARGS=mv /tmp/tmp.d8fwosyqRJ/debezium-connector-postgres /usr/share/confluent-hub-components/debezium-connector-postgres
[2026-02-05 03:43:14 UTC] USER=www-data EUID=0 PID=3567654 ACTION=fsop ARGS=chown -R kafka:kafka /usr/share/confluent-hub-components/debezium-connector-postgres
═══════════════════════════════════════════════════════════════════
[OK] Debezium PostgreSQL connector installed successfully!
Debezium: 3.4.1.Final
pgjdbc: 42.7.9
═══════════════════════════════════════════════════════════════════
═══════════════════════════════════════════════════════════════════
Installing Confluent S3 Sink Connector (Audit Cold Storage)
═══════════════════════════════════════════════════════════════════
[OK] Confluent S3 Sink connector already installed
═══════════════════════════════════════════════════════════════════
[OK] All connectors installed successfully!
- Debezium PostgreSQL: CDC from PostgreSQL
- Confluent S3 Sink: Cold storage for audit retention
═══════════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Step 1 completed: 00-install-debezium-connector.sh
[0;34m[INFO][0m 📦 Step 2/13: kafka setup...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials for secrets vault...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
🧹 Checking for orphaned Kafka processes on ports 9092, 9093, 8083...
⚠️ Found process on port 9092 (PIDs: [2026-02-05 03:43:14 UTC] USER=www-data EUID=0 PID=3567730 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 2>/dev/null || true
886
887
1034), killing...
[2026-02-05 03:43:14 UTC] USER=www-data EUID=0 PID=3567742 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 03:43:14 UTC] USER=www-data EUID=0 PID=3567730 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 2>/dev/null || true
886
887
1034 2>/dev/null || true
/usr/bin/bash: line 2: 886: command not found
/usr/bin/bash: line 3: 887: command not found
⚠️ Found process on port 9093 (PIDs: [2026-02-05 03:43:16 UTC] USER=www-data EUID=0 PID=3567765 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 2>/dev/null || true
886
3565745), killing...
[2026-02-05 03:43:16 UTC] USER=www-data EUID=0 PID=3567784 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 03:43:16 UTC] USER=www-data EUID=0 PID=3567765 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 2>/dev/null || true
886
3565745 2>/dev/null || true
/usr/bin/bash: line 2: 886: command not found
⚠️ Found process on port 8083 (PIDs: [2026-02-05 03:43:17 UTC] USER=www-data EUID=0 PID=3567890 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 2>/dev/null || true
887
889), killing...
[2026-02-05 03:43:17 UTC] USER=www-data EUID=0 PID=3567908 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 03:43:17 UTC] USER=www-data EUID=0 PID=3567890 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 2>/dev/null || true
887
889 2>/dev/null || true
/usr/bin/bash: line 2: 887: command not found
✅ Port cleanup completed
Ensuring KAFKA application environment for coordinator...
[0;34m[INFO][0m Creating KAFKA application environment...
[INFO] 🎯 Custom Environment Creation (Example Wrapper)
[INFO] 📁 Orchestrator Library: /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] 💾 State Directory: /opt/fastorder/bash/scripts/env_app_setup/state
[INFO] 🚀 Calling centralized orchestrator: fo-env create-app
[INFO] 📋 Arguments: --service web --zone universe --branch main --env dev --domain eventbus-web-universe-main-dev-kafka-connect --app kafka-connect
[INFO] Creating application-specific environment configuration
[INFO] Environment ID: web-universe-main-dev
[INFO] Application: kafka-connect
[INFO] Base environment web-universe-main-dev already exists
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Allocated kafka-connect IP: 10.100.1.75
[INFO] Generated domain: eventbus-web-universe-main-dev-kafka-connect.fastorder.com
[INFO] Configuring network interface for kafka-connect IP: 10.100.1.75
[2026-02-05 03:43:19 UTC] USER=www-data EUID=0 PID=3568117 ACTION=passthru ARGS=ip addr add 10.100.1.75/32 dev eth0 label eth0:75
[ OK ] Configured kafka-connect IP 10.100.1.75 on interface eth0:75
[INFO] Creating systemd service for kafka-connect IP persistence...
[2026-02-05 03:43:19 UTC] USER=www-data EUID=0 PID=3568136 ACTION=passthru ARGS=systemctl daemon-reload
[ OK ] kafka-connect IP will persist across reboots
[INFO] Updating topology with application-specific configuration...
[ OK ] Topology updated with application-specific configuration
[INFO] Binding kafka-connect IP to domain: 10.100.1.75 -> eventbus-web-universe-main-dev-kafka-connect.fastorder.com
[ OK ] Successfully bound eventbus-web-universe-main-dev-kafka-connect.fastorder.com to 10.100.1.75
[ OK ] Domain correctly mapped
[ OK ] Application environment created successfully!
[INFO]
[INFO] Application Details:
[INFO] Environment ID: web-universe-main-dev
[INFO] Application: kafka-connect
[INFO] IP: 10.100.1.75
[INFO] Domain: eventbus-web-universe-main-dev-kafka-connect.fastorder.com
[INFO]
[INFO] To use this application:
[INFO] source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO] init_environment kafka-connect
[INFO] echo $VM_IP # Returns: 10.100.1.75
[ OK ] 🎉 Environment creation completed successfully!
[INFO] 📋 What happened:
[INFO] ✅ Called centralized orchestrator at /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] ✅ All topology.json management handled centrally
[INFO] ✅ Application-specific IP and domain configured
[INFO] ✅ Network interface configured and made persistent
[INFO] ✅ Domain binding added to /etc/hosts (if not skipped)
[INFO] 🔧 To use the centralized orchestrator directly:
[INFO] # Add orchestrator to PATH
[INFO] export PATH="/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/bin:$PATH"
[INFO] # Then call directly
[INFO] fo-env create-app --service auth --zone uae --env dev --app redis
[INFO] 📚 For more orchestrator commands:
[INFO] fo-env --help
[0;32mCreated KAFKA environment: eventbus-web-universe-main-dev-kafka-connect.fastorder.com (10.100.1.75)[0m
Ensuring KAFKA_BROKER_IP application environment for coordinator...
[0;34m[INFO][0m Creating KAFKA application environment...
[INFO] 🎯 Custom Environment Creation (Example Wrapper)
[INFO] 📁 Orchestrator Library: /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] 💾 State Directory: /opt/fastorder/bash/scripts/env_app_setup/state
[INFO] 🚀 Calling centralized orchestrator: fo-env create-app
[INFO] 📋 Arguments: --service web --zone universe --branch main --env dev --domain eventbus-web-universe-main-dev-kafka-broker-01 --app kafka-broker
[INFO] Creating application-specific environment configuration
[INFO] Environment ID: web-universe-main-dev
[INFO] Application: kafka-broker
[INFO] Base environment web-universe-main-dev already exists
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Allocated kafka-broker IP: 10.100.1.225
[INFO] Generated domain: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com
[INFO] Configuring network interface for kafka-broker IP: 10.100.1.225
[2026-02-05 03:43:22 UTC] USER=www-data EUID=0 PID=3568740 ACTION=passthru ARGS=ip addr add 10.100.1.225/32 dev eth0 label eth0:225
[ OK ] Configured kafka-broker IP 10.100.1.225 on interface eth0:225
[INFO] Creating systemd service for kafka-broker IP persistence...
[2026-02-05 03:43:22 UTC] USER=www-data EUID=0 PID=3568760 ACTION=passthru ARGS=systemctl daemon-reload
[ OK ] kafka-broker IP will persist across reboots
[INFO] Updating topology with application-specific configuration...
[ OK ] Topology updated with application-specific configuration
[INFO] Binding kafka-broker IP to domain: 10.100.1.225 -> eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com
[ OK ] Successfully bound eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com to 10.100.1.225
[ OK ] Domain correctly mapped
[ OK ] Application environment created successfully!
[INFO]
[INFO] Application Details:
[INFO] Environment ID: web-universe-main-dev
[INFO] Application: kafka-broker
[INFO] IP: 10.100.1.225
[INFO] Domain: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com
[INFO]
[INFO] To use this application:
[INFO] source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO] init_environment kafka-broker
[INFO] echo $VM_IP # Returns: 10.100.1.225
[ OK ] 🎉 Environment creation completed successfully!
[INFO] 📋 What happened:
[INFO] ✅ Called centralized orchestrator at /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] ✅ All topology.json management handled centrally
[INFO] ✅ Application-specific IP and domain configured
[INFO] ✅ Network interface configured and made persistent
[INFO] ✅ Domain binding added to /etc/hosts (if not skipped)
[INFO] 🔧 To use the centralized orchestrator directly:
[INFO] # Add orchestrator to PATH
[INFO] export PATH="/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/bin:$PATH"
[INFO] # Then call directly
[INFO] fo-env create-app --service auth --zone uae --env dev --app redis
[INFO] 📚 For more orchestrator commands:
[INFO] fo-env --help
[0;32mCreated KAFKA_BROKER_DOMAIN environment: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com (10.100.1.225)[0m
[0;34m[INFO][0m Kafka Broker IP: 10.100.1.225
[0;34m[INFO][0m Kafka Connect IP: 10.100.1.75
[0;34m[INFO][0m Registered /etc/hosts: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com -> 10.100.1.225
[0;34m[INFO][0m Registered /etc/hosts: eventbus-web-universe-main-dev-kafka-connect.fastorder.com -> 10.100.1.75
🔐 Initializing keystore passwords...
[0;34m[INFO][0m 🔍 Checking secrets backend (provider: aws)...
[0;33m⚠️ No existing passwords found[0m
[0;34m[INFO][0m 🔐 Generating new random passwords...
[0;32m✅ Generated new passwords[0m
[0;34m[INFO][0m 🔐 Vaulting kafka passwords to remote backend...
[0;32m✅ Passwords vaulted to remote backend[0m
✅ Keystore passwords initialized
- Keystore password: HGvJkWmj... (32 chars)
- Truststore password: sZRdI2nT... (32 chars)
[0;34m[INFO][0m 🔐 Vaulting kafka passwords to remote backend...
[0;32m✅ Passwords vaulted to remote backend[0m
✅ Kafka keystore passwords saved to AWS Secrets Manager
[INFO] Generating for: web-universe-main-dev (host=eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com ip=10.100.1.225)
[2026-02-05 03:43:34 UTC] USER=www-data EUID=0 PID=3569354 ACTION=fsop ARGS=rm -rf /opt/kafka/secrets/web-universe-main-dev/coordinator /data/kafka/web-universe-main-dev/coordinator
[2026-02-05 03:43:34 UTC] USER=www-data EUID=0 PID=3569363 ACTION=fsop ARGS=mkdir -p /opt/kafka/secrets/web-universe-main-dev/coordinator /opt/kafka/config/web-universe-main-dev/coordinator /opt/kafka/secrets/web-universe-main-dev/coordinator/pem /data/kafka/web-universe-main-dev_coordinator-data
[2026-02-05 03:43:34 UTC] USER=www-data EUID=0 PID=3569372 ACTION=fsop ARGS=chown -R kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator
[2026-02-05 03:43:35 UTC] USER=www-data EUID=0 PID=3569381 ACTION=fsop ARGS=chown -R kafka:kafka /opt/kafka/config/web-universe-main-dev/coordinator /data/kafka/web-universe-main-dev_coordinator-data
[2026-02-05 03:43:35 UTC] USER=www-data EUID=0 PID=3569391 ACTION=fsop ARGS=chmod 770 /opt/kafka/config/web-universe-main-dev/coordinator /data/kafka/web-universe-main-dev_coordinator-data
[2026-02-05 03:43:35 UTC] USER=www-data EUID=0 PID=3569400 ACTION=fsop ARGS=chmod 750 /opt/kafka/secrets/web-universe-main-dev/coordinator
[2026-02-05 03:43:35 UTC] USER=www-data EUID=0 PID=3569409 ACTION=fsop ARGS=chmod 750 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem
[2026-02-05 03:43:35 UTC] USER=www-data EUID=0 PID=3569419 ACTION=fsop ARGS=chmod 700 /tmp/fo-tls.QkarjM
[2026-02-05 03:43:35 UTC] USER=www-data EUID=0 PID=3569428 ACTION=fsop ARGS=chmod 755 /tmp/fo-tls.QkarjM
[2026-02-05 03:43:35 UTC] USER=www-data EUID=0 PID=3569437 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/fo-tls.QkarjM/ra_root.crt
[2026-02-05 03:43:35 UTC] USER=www-data EUID=0 PID=3569446 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/fo-tls.QkarjM/ra_root.key
[2026-02-05 03:43:35 UTC] USER=www-data EUID=0 PID=3569455 ACTION=fsop ARGS=chmod 644 /tmp/fo-tls.QkarjM/ra_root.crt
[2026-02-05 03:43:35 UTC] USER=www-data EUID=0 PID=3569464 ACTION=fsop ARGS=chmod 644 /tmp/fo-tls.QkarjM/ra_root.key
Certificate was added to keystore
[2026-02-05 03:43:35 UTC] USER=www-data EUID=0 PID=3569504 ACTION=fsop ARGS=mv /tmp/fo-tls.QkarjM/truststore.jks /opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks
[2026-02-05 03:43:36 UTC] USER=www-data EUID=0 PID=3569515 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks
[2026-02-05 03:43:36 UTC] USER=www-data EUID=0 PID=3569528 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks
Generating 4,096 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 825 days
for: CN=eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com, OU=Kafka Broker, O=FastOrder, C=AE
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /tmp/fo-tls.QkarjM/kafka.server.keystore.jks -destkeystore /tmp/fo-tls.QkarjM/kafka.server.keystore.jks -deststoretype pkcs12".
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /tmp/fo-tls.QkarjM/kafka.server.keystore.jks -destkeystore /tmp/fo-tls.QkarjM/kafka.server.keystore.jks -deststoretype pkcs12".
Certificate request self-signature ok
subject=C = AE, O = FastOrder, OU = Kafka Broker, CN = eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com
Certificate was added to keystore
Warning:
Certificate reply was installed in keystore
Warning:
[2026-02-05 03:43:40 UTC] USER=www-data EUID=0 PID=3569750 ACTION=fsop ARGS=mv /tmp/fo-tls.QkarjM/kafka.server.keystore.jks /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 03:43:40 UTC] USER=www-data EUID=0 PID=3569759 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 03:43:40 UTC] USER=www-data EUID=0 PID=3569768 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.server.keystore.jks
Generating 4,096 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 825 days
for: CN=eventbus-web-universe-main-dev-kafka-connect.fastorder.com, OU=Kafka Connect REST, O=FastOrder, C=AE
Certificate request self-signature ok
subject=C = AE, O = FastOrder, OU = Kafka Connect REST, CN = eventbus-web-universe-main-dev-kafka-connect.fastorder.com
Certificate was added to keystore
Certificate reply was installed in keystore
[2026-02-05 03:43:45 UTC] USER=www-data EUID=0 PID=3570090 ACTION=fsop ARGS=mv /tmp/fo-tls.QkarjM/connect-rest.keystore.p12 /opt/kafka/secrets/web-universe-main-dev/coordinator/connect-rest.keystore.p12
[2026-02-05 03:43:45 UTC] USER=www-data EUID=0 PID=3570099 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/connect-rest.keystore.p12
[2026-02-05 03:43:45 UTC] USER=www-data EUID=0 PID=3570108 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/web-universe-main-dev/coordinator/connect-rest.keystore.p12
Certificate request self-signature ok
subject=CN = kafka-client-web-universe-main-dev, OU = Kafka Client, O = FastOrder, C = AE
[2026-02-05 03:43:45 UTC] USER=www-data EUID=0 PID=3570128 ACTION=fsop ARGS=cp /tmp/fo-tls.QkarjM/ra_root.crt /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem
[2026-02-05 03:43:45 UTC] USER=www-data EUID=0 PID=3570139 ACTION=fsop ARGS=cp /tmp/fo-tls.QkarjM/client-key.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 03:43:45 UTC] USER=www-data EUID=0 PID=3570150 ACTION=fsop ARGS=cp /tmp/fo-tls.QkarjM/client-cert.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
[2026-02-05 03:43:45 UTC] USER=www-data EUID=0 PID=3570161 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
[2026-02-05 03:43:45 UTC] USER=www-data EUID=0 PID=3570170 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-cert.pem
[2026-02-05 03:43:45 UTC] USER=www-data EUID=0 PID=3570179 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client-key.pem
[2026-02-05 03:43:45 UTC] USER=www-data EUID=0 PID=3570189 ACTION=fsop ARGS=mv /tmp/fo-tls.QkarjM/kafka.client.keystore.p12 /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.client.keystore.p12
[2026-02-05 03:43:45 UTC] USER=www-data EUID=0 PID=3570198 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.client.keystore.p12
🔐 Ensuring kafka user has access to PostgreSQL certificates...
✅ kafka is already in postgres group
🧹 Cleaning up conflicting services and processes on Kafka ports on 10.100.1.225...
🔪 Killing processes on 10.100.1.225:8083: [2026-02-05 03:43:45 UTC] USER=www-data EUID=0 PID=3570251 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:8083 -t 2>/dev/null || true
[2026-02-05 03:43:46 UTC] USER=www-data EUID=0 PID=3570267 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 03:43:45 UTC] USER=www-data EUID=0 PID=3570251 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:8083 -t 2>/dev/null || true
🔪 Killing processes on 10.100.1.225:9092: [2026-02-05 03:43:46 UTC] USER=www-data EUID=0 PID=3570278 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:9092 -t 2>/dev/null || true
[2026-02-05 03:43:46 UTC] USER=www-data EUID=0 PID=3570292 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 03:43:46 UTC] USER=www-data EUID=0 PID=3570278 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:9092 -t 2>/dev/null || true
🔪 Killing processes on 10.100.1.225:9093: [2026-02-05 03:43:46 UTC] USER=www-data EUID=0 PID=3570303 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:9093 -t 2>/dev/null || true
[2026-02-05 03:43:46 UTC] USER=www-data EUID=0 PID=3570321 ACTION=passthru ARGS=bash -c kill -9 [2026-02-05 03:43:46 UTC] USER=www-data EUID=0 PID=3570303 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.225:9093 -t 2>/dev/null || true
✅ Port cleanup completed
🔧 Checking for Kafka Connect internal topics with incorrect cleanup policy...
ℹ️ Kafka broker not running, skipping topic cleanup policy fix
🔧 Creating environment-specific systemd units...
🔧 Writing client properties to /etc/kafka/client-web-universe-main-dev-coordinator.properties ...
[2026-02-05 03:43:46 UTC] USER=www-data EUID=0 PID=3570360 ACTION=fsop ARGS=chown root:kafka /etc/kafka/client-web-universe-main-dev-coordinator.properties
[2026-02-05 03:43:46 UTC] USER=www-data EUID=0 PID=3570369 ACTION=fsop ARGS=chmod 0644 /etc/kafka/client-web-universe-main-dev-coordinator.properties
🔧 Creating PEM certificates for PHP mTLS access...
[2026-02-05 03:43:46 UTC] USER=www-data EUID=0 PID=3570378 ACTION=passthru ARGS=bash -c openssl pkcs12 -in '/opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.client.keystore.p12' -clcerts -nokeys -passin pass:'HGvJkWmjjIaZzWVQzIjopYiQoGhZCsRH' -out '/opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.crt' 2>/dev/null
[2026-02-05 03:43:46 UTC] USER=www-data EUID=0 PID=3570388 ACTION=passthru ARGS=bash -c openssl pkcs12 -in '/opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.client.keystore.p12' -nocerts -nodes -passin pass:'HGvJkWmjjIaZzWVQzIjopYiQoGhZCsRH' -out '/opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.key' 2>/dev/null
[2026-02-05 03:43:46 UTC] USER=www-data EUID=0 PID=3570398 ACTION=passthru ARGS=bash -c keytool -exportcert -alias fastorder-ra-root -keystore '/opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks' -storepass 'sZRdI2nTnX4yhzopYtu0ttl9GtAWGH7c' -rfc -file '/opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.crt' 2>/dev/null
[2026-02-05 03:43:47 UTC] USER=www-data EUID=0 PID=3570436 ACTION=fsop ARGS=chown root:kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.crt /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.key /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.crt
[2026-02-05 03:43:47 UTC] USER=www-data EUID=0 PID=3570445 ACTION=fsop ARGS=chmod 0644 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.crt /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/ca.crt
[2026-02-05 03:43:47 UTC] USER=www-data EUID=0 PID=3570454 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/web-universe-main-dev/coordinator/pem/client.key
🔐 Creating connector secrets file for FileConfigProvider...
[2026-02-05 03:43:47 UTC] USER=www-data EUID=0 PID=3570484 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/connector-secrets.properties
[2026-02-05 03:43:47 UTC] USER=www-data EUID=0 PID=3570493 ACTION=fsop ARGS=chmod 0600 /opt/kafka/secrets/web-universe-main-dev/coordinator/connector-secrets.properties
✅ Connector secrets file created: /opt/kafka/secrets/web-universe-main-dev/coordinator/connector-secrets.properties
FileConfigProvider syntax: ${file:/opt/kafka/secrets/web-universe-main-dev/coordinator/connector-secrets.properties:key_name}
🔧 Creating Canary Event timer for pipeline verification...
[2026-02-05 03:43:47 UTC] USER=www-data EUID=0 PID=3570526 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 03:43:48 UTC] USER=www-data EUID=0 PID=3570579 ACTION=passthru ARGS=systemctl enable kafka-canary-web-universe-main-dev.timer
Created symlink /etc/systemd/system/timers.target.wants/kafka-canary-web-universe-main-dev.timer → /etc/systemd/system/kafka-canary-web-universe-main-dev.timer.
[2026-02-05 03:43:48 UTC] USER=www-data EUID=0 PID=3570629 ACTION=passthru ARGS=systemctl start kafka-canary-web-universe-main-dev.timer
✅ Canary timer installed: kafka-canary-web-universe-main-dev.timer (every 5 minutes)
[2026-02-05 03:43:48 UTC] USER=www-data EUID=0 PID=3570640 ACTION=passthru ARGS=systemctl daemon-reload
Failed to print table: Broken pipe
[2026-02-05 03:43:56 UTC] USER=www-data EUID=0 PID=3570856 ACTION=passthru ARGS=systemctl mask kafka-broker
🔒 Adjusting group ownership and permissions ...
[2026-02-05 03:43:56 UTC] USER=www-data EUID=0 PID=3570989 ACTION=fsop ARGS=chown :kafka /opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 03:43:56 UTC] USER=www-data EUID=0 PID=3571003 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.server.keystore.jks
[2026-02-05 03:43:56 UTC] USER=www-data EUID=0 PID=3571015 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.client.keystore.p12
[2026-02-05 03:43:57 UTC] USER=www-data EUID=0 PID=3571028 ACTION=fsop ARGS=chown root:kafka /etc/kafka/client-web-universe-main-dev-coordinator.properties
[2026-02-05 03:43:57 UTC] USER=www-data EUID=0 PID=3571037 ACTION=fsop ARGS=chmod 0644 /etc/kafka/client-web-universe-main-dev-coordinator.properties
✅ Kafka configuration complete for web-universe-main-dev_coordinator
Broker ID : 89
Broker keystore : /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.server.keystore.jks
REST keystore : /opt/kafka/secrets/web-universe-main-dev/coordinator/connect-rest.keystore.p12
Truststore : /opt/kafka/secrets/web-universe-main-dev/coordinator/truststore.jks
Client PKCS12 : /opt/kafka/secrets/web-universe-main-dev/coordinator/kafka.client.keystore.p12
Data directory : /data/kafka/web-universe-main-dev_coordinator-data
Server config : /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
Connect config : /opt/kafka/config/web-universe-main-dev/coordinator/connect-distributed.properties
CLI client config : /etc/kafka/client-web-universe-main-dev-coordinator.properties
🎯 Next step: Run 03-restart-kafka-related-services.sh to start services
[0;32m[OK][0m ✅ Step 2 completed: 01-kafka-setup.sh
[0;34m[INFO][0m 📦 Step 3/13: metadata...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] 🎯 Kafka metadata mode: kraft
╔════════════════════════════════════════════════════════════════════╗
║ Kafka Metadata Layer Setup ║
╚════════════════════════════════════════════════════════════════════╝
Environment : web-universe-main-dev
Service : web
Zone : universe
Branch : main
Environment : dev
VM IP : 142.93.238.16
Metadata Mode : kraft
📋 KRaft Mode (Modern)
────────────────────────────────────────────────────────────────
✅ No ZooKeeper dependency
✅ Faster metadata operations
✅ Simplified architecture
✅ Recommended for new deployments
⚠️ Requires Kafka 3.3+ in production
════════════════════════════════════════════════════════════════════
[INFO] 🚀 Executing KRaft setup script...
[INFO] Script: /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps/metadata/kraft.sh
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 03:43:57 UTC] USER=www-data EUID=0 PID=3571107 ACTION=fsop ARGS=mkdir -p /data/kafka/web-universe-main-dev_coordinator-meta /opt/kafka/config/web-universe-main-dev/coordinator /data/kafka/web-universe-main-dev_coordinator-data
[2026-02-05 03:43:57 UTC] USER=www-data EUID=0 PID=3571116 ACTION=fsop ARGS=chown -R kafka:kafka /data/kafka/web-universe-main-dev_coordinator-meta /opt/kafka/config/web-universe-main-dev/coordinator /data/kafka/web-universe-main-dev_coordinator-data
[INFO] Adding eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com to /etc/hosts -> 10.100.1.225
[INFO] Adding eventbus-web-universe-main-dev-kafka-connect.fastorder.com to /etc/hosts -> 10.100.1.75
[INFO] Setting up KRaft for: web-universe-main-dev (host=eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com ip=10.100.1.225)
[2026-02-05 03:43:58 UTC] USER=www-data EUID=0 PID=3571149 ACTION=fsop ARGS=mkdir -p /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev
[INFO] Generated cluster.id=uBayQf0-RSyd3l5SczIXzA
🔧 Configuring Kafka for KRaft mode...
[2026-02-05 03:44:00 UTC] USER=www-data EUID=0 PID=3571705 ACTION=fsop ARGS=test -f /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 03:44:00 UTC] USER=www-data EUID=0 PID=3571716 ACTION=fsop ARGS=test -r /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 03:44:00 UTC] USER=www-data EUID=0 PID=3571725 ACTION=fsop ARGS=sed -i /^zookeeper\.connect=/d /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 03:44:00 UTC] USER=www-data EUID=0 PID=3571735 ACTION=passthru ARGS=bash -c grep -q '^process.roles=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 03:44:01 UTC] USER=www-data EUID=0 PID=3571759 ACTION=passthru ARGS=bash -c grep -q '^node.id=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 03:44:01 UTC] USER=www-data EUID=0 PID=3571799 ACTION=passthru ARGS=bash -c grep -q '^broker.id=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 03:44:01 UTC] USER=www-data EUID=0 PID=3571808 ACTION=fsop ARGS=sed -i s|^broker.id=.*|broker.id=1| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 03:44:01 UTC] USER=www-data EUID=0 PID=3571818 ACTION=passthru ARGS=bash -c grep -q '^controller.listener.names=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 03:44:01 UTC] USER=www-data EUID=0 PID=3571837 ACTION=passthru ARGS=bash -c grep -q '^controller.quorum.voters=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 03:44:01 UTC] USER=www-data EUID=0 PID=3571857 ACTION=passthru ARGS=bash -c grep -q '^metadata.log.dir=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 03:44:01 UTC] USER=www-data EUID=0 PID=3571877 ACTION=passthru ARGS=bash -c grep -q '^log.dirs=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 03:44:01 UTC] USER=www-data EUID=0 PID=3571886 ACTION=fsop ARGS=sed -i s|^log.dirs=.*|log.dirs=/data/kafka/web-universe-main-dev_coordinator-data| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 03:44:01 UTC] USER=www-data EUID=0 PID=3571895 ACTION=passthru ARGS=bash -c grep -q '^listeners=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 03:44:01 UTC] USER=www-data EUID=0 PID=3571909 ACTION=passthru ARGS=bash -c grep -q 'CONTROLLER://' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 03:44:01 UTC] USER=www-data EUID=0 PID=3571922 ACTION=fsop ARGS=sed -i s|^listeners=.*|listeners=SSL://10.100.1.225:9092,CONTROLLER://10.100.1.225:9093| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 03:44:01 UTC] USER=www-data EUID=0 PID=3571932 ACTION=passthru ARGS=bash -c grep -q '^advertised.listeners=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 03:44:01 UTC] USER=www-data EUID=0 PID=3571941 ACTION=fsop ARGS=sed -i s|^advertised.listeners=.*|advertised.listeners=SSL://eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 03:44:02 UTC] USER=www-data EUID=0 PID=3571950 ACTION=passthru ARGS=bash -c grep -q '^listener.security.protocol.map=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 03:44:02 UTC] USER=www-data EUID=0 PID=3571961 ACTION=fsop ARGS=sed -i s|^listener.security.protocol.map=.*|listener.security.protocol.map=SSL:SSL,CONTROLLER:PLAINTEXT| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 03:44:02 UTC] USER=www-data EUID=0 PID=3571979 ACTION=fsop ARGS=sed -i s|^inter.broker.listener.name=.*|inter.broker.listener.name=SSL| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 03:44:02 UTC] USER=www-data EUID=0 PID=3571988 ACTION=passthru ARGS=bash -c grep -q '^offsets.topic.replication.factor=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 03:44:02 UTC] USER=www-data EUID=0 PID=3571998 ACTION=fsop ARGS=sed -i s|^offsets.topic.replication.factor=.*|offsets.topic.replication.factor=1| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 03:44:02 UTC] USER=www-data EUID=0 PID=3572007 ACTION=passthru ARGS=bash -c grep -q '^transaction.state.log.replication.factor=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 03:44:02 UTC] USER=www-data EUID=0 PID=3572016 ACTION=fsop ARGS=sed -i s|^transaction.state.log.replication.factor=.*|transaction.state.log.replication.factor=1| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 03:44:02 UTC] USER=www-data EUID=0 PID=3572029 ACTION=passthru ARGS=bash -c grep -q '^transaction.state.log.min.isr=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
[2026-02-05 03:44:02 UTC] USER=www-data EUID=0 PID=3572045 ACTION=fsop ARGS=sed -i s|^transaction.state.log.min.isr=.*|transaction.state.log.min.isr=1| /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 03:44:02 UTC] USER=www-data EUID=0 PID=3572056 ACTION=passthru ARGS=bash -c grep -q '^min.insync.replicas=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
✅ KRaft configuration applied to server.properties
[2026-02-05 03:44:02 UTC] USER=www-data EUID=0 PID=3572076 ACTION=fsop ARGS=test -f /data/kafka/web-universe-main-dev_coordinator-meta/meta.properties
[INFO] Formatting storage directories for coordinator (cluster.id=[2026-02-05 03:44:00 UTC] USER=www-data EUID=0 PID=3571696 ACTION=passthru ARGS=bash -c cat /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/kafka_kraft_cluster_id
uBayQf0-RSyd3l5SczIXzA)
[2026-02-05 03:44:02 UTC] USER=www-data EUID=0 PID=3572086 ACTION=passthru ARGS=sudo -u kafka /opt/kafka/bin/kafka-storage.sh format --config /opt/kafka/config/web-universe-main-dev/coordinator/server.properties --cluster-id [2026-02-05 03:44:00 UTC] USER=www-data EUID=0 PID=3571696 ACTION=passthru ARGS=bash -c cat /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/kafka_kraft_cluster_id
uBayQf0-RSyd3l5SczIXzA --ignore-formatted
Formatting metadata directory /data/kafka/web-universe-main-dev_coordinator-meta with metadata.version 3.9-IV0.
Formatting data directory /data/kafka/web-universe-main-dev_coordinator-data with metadata.version 3.9-IV0.
[INFO] Formatted: /data/kafka/web-universe-main-dev_coordinator-meta and /data/kafka/web-universe-main-dev_coordinator-data
🔧 Creating/refreshing KRaft systemd unit...
[2026-02-05 03:44:06 UTC] USER=www-data EUID=0 PID=3572565 ACTION=fsop ARGS=sed -i s|\\$MAINPID|$MAINPID|g /etc/systemd/system/confluent-kraft-web-universe-main-dev_coordinator.service
[2026-02-05 03:44:06 UTC] USER=www-data EUID=0 PID=3572575 ACTION=passthru ARGS=systemctl daemon-reload
✅ Ensured confluent-kraft-web-universe-main-dev_coordinator.service
🛑 Stopping legacy ZooKeeper-mode services and current KRaft instance...
🧹 Cleaning up rogue Kafka processes...
🧹 Killing any processes holding Kafka ports 9092, 9093...
🔪 Killing processes on port 9092: 886
887
1034
[2026-02-05 03:44:07 UTC] USER=www-data EUID=0 PID=3572714 ACTION=passthru ARGS=bash -c kill -9 886
[2026-02-05 03:44:07 UTC] USER=www-data EUID=0 PID=3572723 ACTION=passthru ARGS=bash -c kill -9 887
[2026-02-05 03:44:07 UTC] USER=www-data EUID=0 PID=3572735 ACTION=passthru ARGS=bash -c kill -9 1034
🔪 Killing processes on port 9093: 3565745
[2026-02-05 03:44:09 UTC] USER=www-data EUID=0 PID=3572814 ACTION=passthru ARGS=bash -c kill -9 3565745
🔪 Killing processes on port 8083: 889
[2026-02-05 03:44:12 UTC] USER=www-data EUID=0 PID=3572887 ACTION=passthru ARGS=bash -c kill -9 889
✅ Legacy services stopped and rogue processes cleaned
🔓 Removing stale lock files...
[2026-02-05 03:44:17 UTC] USER=www-data EUID=0 PID=3573071 ACTION=fsop ARGS=test -f /data/kafka/web-universe-main-dev_coordinator-meta/.lock
✅ Lock file check complete
🚀 Starting confluent-kraft-web-universe-main-dev_coordinator.service ...
[2026-02-05 03:44:17 UTC] USER=www-data EUID=0 PID=3573092 ACTION=passthru ARGS=systemctl enable confluent-kraft-web-universe-main-dev_coordinator.service
Created symlink /etc/systemd/system/multi-user.target.wants/confluent-kraft-web-universe-main-dev_coordinator.service → /etc/systemd/system/confluent-kraft-web-universe-main-dev_coordinator.service.
[2026-02-05 03:44:18 UTC] USER=www-data EUID=0 PID=3573146 ACTION=passthru ARGS=systemctl restart confluent-kraft-web-universe-main-dev_coordinator.service
🔧 Patching shared Connect unit to follow KRaft broker...
[2026-02-05 03:44:21 UTC] USER=www-data EUID=0 PID=3573653 ACTION=fsop ARGS=sed -i -e s|${FULL_ENV}|web-universe-main-dev|g -e s|${IDENTIFIER}|coordinator|g -e s|${CONFIG_DIR}|/opt/kafka/config/web-universe-main-dev/coordinator|g /etc/systemd/system/confluent-connect-web-universe-main-dev_coordinator.service
[2026-02-05 03:44:21 UTC] USER=www-data EUID=0 PID=3573676 ACTION=fsop ARGS=sed -i s|\\$MAINPID|$MAINPID|g /etc/systemd/system/confluent-connect-web-universe-main-dev_coordinator.service
[2026-02-05 03:44:21 UTC] USER=www-data EUID=0 PID=3573692 ACTION=fsop ARGS=sed -i s|^After=.*|After=network-online.target confluent-kraft-web-universe-main-dev_coordinator.service| /etc/systemd/system/confluent-connect-web-universe-main-dev_coordinator.service
[2026-02-05 03:44:21 UTC] USER=www-data EUID=0 PID=3573703 ACTION=fsop ARGS=sed -i s|^Wants=.*|Wants=confluent-kraft-web-universe-main-dev_coordinator.service| /etc/systemd/system/confluent-connect-web-universe-main-dev_coordinator.service
[2026-02-05 03:44:21 UTC] USER=www-data EUID=0 PID=3573715 ACTION=fsop ARGS=sed -i s|^ExecStart=.*|ExecStart=/opt/kafka/bin/connect-distributed.sh /opt/kafka/config/web-universe-main-dev/coordinator/connect-distributed.properties| /etc/systemd/system/confluent-connect-web-universe-main-dev_coordinator.service
[2026-02-05 03:44:21 UTC] USER=www-data EUID=0 PID=3573727 ACTION=passthru ARGS=systemctl daemon-reload
✅ Connect unit patched
[2026-02-05 03:44:22 UTC] USER=www-data EUID=0 PID=3573788 ACTION=fsop ARGS=test -f /opt/kafka/config/web-universe-main-dev/coordinator/connect-distributed.properties
[2026-02-05 03:44:22 UTC] USER=www-data EUID=0 PID=3573797 ACTION=fsop ARGS=ln -sf /opt/kafka/config/web-universe-main-dev/coordinator/connect-distributed.properties /opt/kafka/config/connect-distributed.properties
⏳ Waiting for broker coordinator on SSL://eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092 ...
⏳ Waiting for KRaft broker... (attempt 1, 0s/600s)
Debug: Last error was: [2026-02-05 03:44:22 UTC] USER=www-data EUID=0 PID=3573809 ACTION=passthru ARGS=bash -c timeout 5 sudo -u kafka /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server 'eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092' --command-config '/etc/kafka/client-web-universe-main-dev-coordinator.properties' describe --status
⏳ Still waiting... (attempt 10, 59s/600s)
✅ coordinator responded after 91s (attempt 15)
---- server.properties (key lines) ----
[2026-02-05 03:46:24 UTC] USER=www-data EUID=0 PID=3592313 ACTION=passthru ARGS=bash -c grep -E '^(listeners|advertised\.listeners|process\.roles|controller\.quorum\.voters|controller\.listener\.names|inter\.broker\.listener\.name|log\.dirs|metadata\.log\.dir)=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties'
listeners=SSL://10.100.1.225:9092,CONTROLLER://10.100.1.225:9093
advertised.listeners=SSL://eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092
inter.broker.listener.name=SSL
log.dirs=/data/kafka/web-universe-main-dev_coordinator-data
process.roles=broker,controller
controller.listener.names=CONTROLLER
controller.quorum.voters=1@10.100.1.225:9093
metadata.log.dir=/data/kafka/web-universe-main-dev_coordinator-meta
---------------------------------------
✅ KRaft setup complete for web-universe-main-dev_coordinator
server.properties : /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
data dir : /data/kafka/web-universe-main-dev_coordinator-data
meta dir : /data/kafka/web-universe-main-dev_coordinator-meta
systemd unit : confluent-kraft-web-universe-main-dev_coordinator.service
🔧 Kafka Configuration Modified:
✓ process.roles, node.id, controller.quorum.voters, controller.listener.names
✓ listeners (SSL + CONTROLLER) and advertised.listeners (FQDN fallback to IP)
✓ listener.security.protocol.map, inter.broker.listener.name
✓ log.dirs -> /data/kafka/web-universe-main-dev_coordinator-data, metadata.log.dir -> /data/kafka/web-universe-main-dev_coordinator-meta
✓ removed zookeeper.connect (if present)
✓ created/refreshed dedicated KRaft systemd unit
✓ patched shared Connect unit to follow KRaft broker
✓ symlinked /opt/kafka/config/web-universe-main-dev/coordinator/connect-distributed.properties -> /opt/kafka/config/connect-distributed.properties (compat)
🔎 Check quorum:
/opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092 --command-config /etc/kafka/client-web-universe-main-dev-coordinator.properties describe --status
📋 Next steps:
1) Review KRaft config: sudo grep -E 'process.roles|node.id|controller|listeners|advertised.listeners|log.dirs|metadata.log.dir' /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
2) Verify topics: /opt/kafka/bin/kafka-topics.sh --bootstrap-server eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092 --command-config /etc/kafka/client-web-universe-main-dev-coordinator.properties --list
✅ KRaft metadata layer setup completed successfully
Next steps:
1. Verify KRaft quorum status
2. Create Kafka topics
3. Configure Kafka Connect
[2026-02-05 03:46:24 UTC] USER=www-data EUID=0 PID=3592323 ACTION=fsop ARGS=mkdir -p /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev
[INFO] Saved metadata mode to: /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/kafka_metadata_mode
════════════════════════════════════════════════════════════════════
✅ Kafka Metadata Layer Setup Complete
Mode : kraft
Environment : web-universe-main-dev
State saved : /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/kafka_metadata_mode
KRaft cluster.id: uBayQf0-RSyd3l5SczIXzA
Verify quorum:
kafka-metadata-quorum.sh --bootstrap-server ... describe
════════════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Step 3 completed: 02-metadata.sh
[0;34m[INFO][0m 📦 Step 4/13: restart kafka related services...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 03:46:25 UTC] USER=www-data EUID=0 PID=3592377 ACTION=fsop ARGS=test -f /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 03:46:25 UTC] USER=www-data EUID=0 PID=3592386 ACTION=passthru ARGS=bash -c grep -E '^[[:space:]]*process\.roles=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties' | grep -Eq '(broker|controller)'
[INFO] 📋 Detected mode from server.properties: kraft
[2026-02-05 03:46:25 UTC] USER=www-data EUID=0 PID=3592473 ACTION=passthru ARGS=systemctl stop confluent-connect-web-universe-main-dev_coordinator.service
[2026-02-05 03:46:26 UTC] USER=www-data EUID=0 PID=3592553 ACTION=passthru ARGS=systemctl stop confluent-kafka-zk-web-universe-main-dev_coordinator.service
[2026-02-05 03:46:26 UTC] USER=www-data EUID=0 PID=3592615 ACTION=passthru ARGS=systemctl stop confluent-zookeeper-web-universe-main-dev_coordinator.service
Failed to stop confluent-zookeeper-web-universe-main-dev_coordinator.service: Unit confluent-zookeeper-web-universe-main-dev_coordinator.service not loaded.
[INFO] 🧹 Removing stale Kafka lock files...
[2026-02-05 03:46:30 UTC] USER=www-data EUID=0 PID=3592775 ACTION=fsop ARGS=rm -f /var/lib/kafka/web-universe-main-dev_coordinator-meta/.lock
[2026-02-05 03:46:30 UTC] USER=www-data EUID=0 PID=3592792 ACTION=fsop ARGS=rm -f /var/lib/kafka/web-universe-main-dev_coordinator-data/.lock
[INFO] 🧹 Cleaning up orphaned processes on Kafka ports...
[2026-02-05 03:46:30 UTC] USER=www-data EUID=0 PID=3592801 ACTION=passthru ARGS=bash -c
for port in 9092 9093 8083 2181; do
pids=$(lsof -ti tcp:$port 2>/dev/null || true)
if [[ -n "$pids" ]]; then
echo " Killing orphaned processes on port $port: $pids"
kill -9 $pids 2>/dev/null || true
sleep 1
fi
done
Killing orphaned processes on port 9092: 1035
3573153
3574069
3574070
3574071
3575241
3575242
Killing orphaned processes on port 9093: 3573626
🚀 Restarting Kafka components…
[INFO] 🚀 starting confluent-kraft-web-universe-main-dev_coordinator.service…
[2026-02-05 03:46:34 UTC] USER=www-data EUID=0 PID=3593103 ACTION=passthru ARGS=systemctl restart confluent-kraft-web-universe-main-dev_coordinator.service
[INFO] 🚀 starting confluent-connect-web-universe-main-dev_coordinator.service…
[2026-02-05 03:46:36 UTC] USER=www-data EUID=0 PID=3593638 ACTION=passthru ARGS=systemctl restart confluent-connect-web-universe-main-dev_coordinator.service
[INFO] ⏳ Waiting for Kafka broker readiness (FQDN: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com, IP: 10.100.1.225) ...
[OK] ✅ Broker ready (attempt 1)
[OK] ✅ Port 9092 listening (Kafka Broker)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (1/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (2/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (3/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (4/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (5/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (6/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (7/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (8/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (9/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (10/40)
[OK] ✅ Port 8083 listening (Kafka Connect REST)
[INFO] ⏳ Waiting for Connect REST at https://eventbus-web-universe-main-dev-kafka-connect.fastorder.com:8083 …
[OK] ✅ Connect REST is up (attempt 1)
📋 Reconciling Connect internal topics…
[ok] connect-configs exists
[ok] connect-offsets exists
[ok] connect-status exists
═══════════════════════════════════════════════════════════════════
KAFKA SUMMARY
═══════════════════════════════════════════════════════════════════
Env: web-universe-main-dev Identifier: coordinator Mode: kraft
Broker Unit : confluent-kraft-web-universe-main-dev_coordinator.service (status: active)
Connect Unit: confluent-connect-web-universe-main-dev_coordinator.service (status: active)
Bootstrap : eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092
Connect URL : https://eventbus-web-universe-main-dev-kafka-connect.fastorder.com:8083
═══════════════════════════════════════════════════════════════════
[OK] ✅ All required services are up.
[0;32m[OK][0m ✅ Step 4 completed: 03-restart-kafka-related-services.sh
[0;34m[INFO][0m 📦 Step 5/13: checking services...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[2026-02-05 03:48:17 UTC] USER=www-data EUID=0 PID=3601575 ACTION=fsop ARGS=test -f /opt/kafka/config/web-universe-main-dev/coordinator/server.properties
[2026-02-05 03:48:17 UTC] USER=www-data EUID=0 PID=3601588 ACTION=passthru ARGS=bash -c grep -E '^[[:space:]]*process\.roles=' '/opt/kafka/config/web-universe-main-dev/coordinator/server.properties' | grep -Eq '(broker|controller)'
[INFO] Detected mode from server.properties: kraft
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 1: Service status
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] confluent-kraft-web-universe-main-dev_coordinator.service status: active
[WARN] confluent-kafka-zk-web-universe-main-dev_coordinator.service present but should be stopped in KRaft
[WARN] confluent-zookeeper-web-universe-main-dev_coordinator.service present but not required in KRaft
[OK] confluent-connect-web-universe-main-dev_coordinator.service status: active
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 2: Port checks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Port 9092 listening (Kafka Broker)
[OK] ✅ Port 8083 listening (Kafka Connect REST)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 3: Broker readiness
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] Broker API responding (attempt 1)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 4: Kafka Connect REST
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] Connect REST responding (attempt 1)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Kafka Services Summary
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment : web-universe-main-dev
Identifier : coordinator
Mode : kraft
Broker Unit : confluent-kraft-web-universe-main-dev_coordinator.service (status: active)
Connect Unit: confluent-connect-web-universe-main-dev_coordinator.service (status: active)
Broker FQDN : eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092
Broker IP : eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092
Connect URL : https://eventbus-web-universe-main-dev-kafka-connect.fastorder.com:8083
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ All required services are reachable.
[INFO] Creating ACLs for Kafka Connect consumer groups...
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
[INFO] Creating ACLs for Connect internal topics...
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
[OK] ✅ Kafka Connect ACLs configured (deny-by-default mode)
[0;32m[OK][0m ✅ Step 5 completed: 04-checking-services.sh
[0;34m[INFO][0m 📦 Step 6/13: create audit topic...
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Creating Kafka Audit Topics
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Environment: web-universe-main-dev
[0;34m[INFO][0m Replication Factor: 1
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 🔍 AUDIT READINESS GATE - Preflight Checks
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m [Gate 1/5] Verifying DNS resolution...
[0;32m[OK][0m ✅ Broker DNS: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com -> 10.100.1.225
[0;32m[OK][0m ✅ Connect DNS: eventbus-web-universe-main-dev-kafka-connect.fastorder.com -> 10.100.1.75
[0;34m[INFO][0m [Gate 2/5] Verifying TLS handshake...
[0;32m[OK][0m ✅ TLS handshake: Broker certificate verified
[0;34m[INFO][0m [Gate 3/5] Verifying Kafka Connect REST API...
[0;32m[OK][0m ✅ Kafka Connect REST: Cluster ID = [2026-02-05 03:44:00 UTC] USER=www-data EUID=0 PID=3571696 ACTION=passthru ARGS=bash -c cat /opt/fastorder/bash/scripts/env_app_setup/state/web-universe-main-dev/kafka_kraft_cluster_id
uBayQf0-RSyd3l5SczIXzA
[0;34m[INFO][0m [Gate 4/5] Verifying required internal topics...
[0;32m[OK][0m ✅ Topic exists: connect-configs
[0;32m[OK][0m ✅ Topic exists: connect-offsets
[0;32m[OK][0m ✅ Topic exists: connect-status
[0;34m[INFO][0m [Gate 5/5] Verifying broker metadata access...
[0;32m[OK][0m ✅ Broker metadata: API versions accessible
[0;32m[OK][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ AUDIT READINESS GATE: ALL CHECKS PASSED
[0;32m[OK][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Waiting for Kafka to be ready...
[0;32m[OK][0m ✅ Kafka is ready
[0;34m[INFO][0m Creating audit topic: audit.events.web.universe.main.dev
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic audit.events.web.universe.main.dev.
[0;32m[OK][0m ✅ Audit topic created: audit.events.web.universe.main.dev
[0;32m[OK][0m ✅ Topic verified: audit.events.web.universe.main.dev
Topic: audit.events.web.universe.main.dev TopicId: _h9pJC4FSPSnOavETTJe4Q PartitionCount: 3 ReplicationFactor: 1 Configs: compression.type=lz4,min.insync.replicas=1,cleanup.policy=delete,segment.bytes=1073741824,retention.ms=7776000000,message.timestamp.type=LogAppendTime,segment.ms=604800000
Topic: audit.events.web.universe.main.dev Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: audit.events.web.universe.main.dev Partition: 1 Leader: 1 Replicas: 1 Isr: 1
Topic: audit.events.web.universe.main.dev Partition: 2 Leader: 1 Replicas: 1 Isr: 1
[0;34m[INFO][0m Creating audit producer credentials...
Completed updating config for user audit-producer-web-universe-main-dev.
[0;32m[OK][0m ✅ Audit producer user created: audit-producer-web-universe-main-dev
[0;34m[INFO][0m Creating ACLs for audit producer...
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.web.universe.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-web-universe-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:audit-producer-web-universe-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-web-universe-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
[0;32m[OK][0m ✅ ACLs configured (producer: write-only, sinks: read-only, immutability: protected)
[0;34m[INFO][0m Storing audit producer credentials in AWS Secrets Manager...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/eventbus/web/universe/main/dev/kafka/audit/producer-X3Fpzs",
"Name": "fastorder/eventbus/web/universe/main/dev/kafka/audit/producer",
"VersionId": "cfb684ec-b9a4-47e9-bc28-496c1ad4dd26"
}
[0;32m[OK][0m ✅ Credentials stored in: fastorder/eventbus/web/universe/main/dev/kafka/audit/producer
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Configuring S3 Sink for Audit Cold Storage
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
PRE audit/
[0;34m[INFO][0m S3 staging bucket exists: fastorder-audit-staging
[0;34m[INFO][0m Data flow: Kafka → fastorder-audit-staging → (replication) → fastorder-audit-immutable
[0;34m[INFO][0m Creating new S3 sink connector...
{"name":"audit-s3-sink-web_universe_main_dev","config":{"connector.class":"io.confluent.connect.s3.S3SinkConnector","tasks.max":"1","topics":"audit.events.web.universe.main.dev","topics.dir":"audit/web-universe-main-dev","s3.bucket.name":"fastorder-audit-staging","s3.region":"me-central-1","s3.part.size":"5242880","flush.size":"1000","rotate.interval.ms":"3600000","rotate.schedule.interval.ms":"86400000","storage.class":"io.confluent.connect.s3.storage.S3Storage","format.class":"io.confluent.connect.s3.format.json.JsonFormat","partitioner.class":"io.confluent.connect.storage.partitioner.TimeBasedPartitioner","path.format":"'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH","partition.duration.ms":"3600000","locale":"en-US","timezone":"UTC","timestamp.extractor":"Record","key.converter":"org.apache.kafka.connect.json.JsonConverter","value.converter":"org.apache.kafka.connect.json.JsonConverter","key.converter.schemas.enable":"false","value.converter.schemas.enable":"false","behavior.on.null.values":"ignore","errors.tolerance":"all","errors.log.enable":"true","errors.log.include.messages":"true","name":"audit-s3-sink-web_universe_main_dev"},"tasks":[],"type":"sink"}[0;32m[OK][0m ✅ S3 Sink connector configured for audit cold storage
[0;34m[INFO][0m Staging Bucket: fastorder-audit-staging (Kafka Connect writes here)
[0;34m[INFO][0m Immutable Bucket: fastorder-audit-immutable (via S3 Replication)
[0;34m[INFO][0m Path: audit/web-universe-main-dev/
[0;34m[INFO][0m Final Retention: WORM-enabled (Object Lock COMPLIANCE mode, 1-year)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Kafka Audit Topic Created Successfully
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Topic: audit.events.web.universe.main.dev
[0;34m[INFO][0m Partitions: 3
[0;34m[INFO][0m Replication Factor: 1
[0;34m[INFO][0m Retention: 90 days
[0;34m[INFO][0m Producer: audit-producer-web-universe-main-dev (write-only)
[0;34m[INFO][0m Application Integration:
[0;34m[INFO][0m - Use credentials from: fastorder/eventbus/web/universe/main/dev/kafka/audit/producer
[0;34m[INFO][0m - Connect to: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092
[0;34m[INFO][0m - Produce to: audit.events.web.universe.main.dev
[0;34m[INFO][0m - Security: SASL_SSL (SCRAM-SHA-512)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 📋 PCI-DSS Compliance Status
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m
[0;34m[INFO][0m ✅ Kafka Hot Storage: 90 days (meets PCI-DSS 3-month immediate access)
[0;34m[INFO][0m ✅ ACL Authorization: deny-by-default (allow.everyone.if.no.acl.found=false)
[0;34m[INFO][0m ✅ Immutability: DENY DELETE/ALTER on audit topic
[0;34m[INFO][0m ✅ S3 Cold Storage: fastorder-audit-immutable (Object Lock COMPLIANCE, 1-year)
[0;34m[INFO][0m
[0;34m[INFO][0m S3 Audit Storage:
[0;34m[INFO][0m Bucket: s3://fastorder-audit-staging
[0;34m[INFO][0m Path: audit/web-universe-main-dev/
[0;34m[INFO][0m Object Lock: COMPLIANCE mode, 1-year retention
[0;34m[INFO][0m Immutability: Objects cannot be deleted or modified for 1 year
[0;34m[INFO][0m
[0;34m[INFO][0m Verify compliance with:
[0;34m[INFO][0m bash 04-eventbus/engine/kafka/steps/11-audit-compliance-check.sh
[0;34m[INFO][0m
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 📋 AWS Roles Anywhere - Credential Refresh Setup
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m
[0;34m[INFO][0m For S3 sink to write to S3, Kafka Connect needs AWS credentials.
[0;34m[INFO][0m Use IAM Roles Anywhere with systemd timer for automatic refresh.
[0;34m[INFO][0m
[0;34m[INFO][0m Required files:
[0;34m[INFO][0m Certificate: /etc/fastorder/rolesanywhere/client-bundle.crt
[0;34m[INFO][0m Private Key: /etc/fastorder/rolesanywhere/client.key
[0;34m[INFO][0m Helper: /usr/local/bin/aws_signing_helper
[0;34m[INFO][0m
[0;34m[INFO][0m Systemd timer: kafka-aws-credential-refresh.timer
[0;34m[INFO][0m Runs every 30 minutes to refresh credentials to /var/lib/kafka/.aws/credentials
[0;34m[INFO][0m
[0;34m[INFO][0m Verify timer is active:
[0;34m[INFO][0m systemctl status kafka-aws-credential-refresh.timer
[0;34m[INFO][0m
[0;34m[INFO][0m Documentation: https://docs.aws.amazon.com/rolesanywhere/latest/userguide/
[0;32m[OK][0m ✅ Step 6 completed: 05-create-audit-topic.sh
[0;34m[INFO][0m 📦 Step 7/13: setup backups...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Kafka Backup Configuration
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Environment: web-universe-main-dev
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[0;34m[INFO][0m 1️⃣ Creating S3 bucket for Kafka backups...
make_bucket failed: s3://fastorder-kafka-backups-web-universe-main-dev An error occurred (AccessDenied) when calling the CreateBucket operation: User: arn:aws:iam::464621692046:user/fo-dev is not authorized to perform: s3:CreateBucket on resource: "arn:aws:s3:::fastorder-kafka-backups-web-universe-main-dev" because no identity-based policy allows the s3:CreateBucket action
An error occurred (NoSuchBucket) when calling the PutBucketVersioning operation: The specified bucket does not exist
Parameter validation failed:
Unknown parameter in LifecycleConfiguration.Rules[0]: "Id", must be one of: Expiration, ID, Prefix, Filter, Status, Transitions, NoncurrentVersionTransitions, NoncurrentVersionExpiration, AbortIncompleteMultipartUpload
[0;32m[OK][0m ✅ S3 bucket created: fastorder-kafka-backups-web-universe-main-dev
[0;34m[INFO][0m 2️⃣ Creating local backup directory...
[2026-02-05 03:49:52 UTC] USER=www-data EUID=0 PID=3606167 ACTION=fsop ARGS=mkdir -p /var/backups/kafka/web-universe-main-dev
[2026-02-05 03:49:52 UTC] USER=www-data EUID=0 PID=3606177 ACTION=fsop ARGS=mkdir -p /var/backups/kafka/web-universe-main-dev/topics
[2026-02-05 03:49:52 UTC] USER=www-data EUID=0 PID=3606190 ACTION=fsop ARGS=mkdir -p /var/backups/kafka/web-universe-main-dev/metadata
[2026-02-05 03:49:52 UTC] USER=www-data EUID=0 PID=3606204 ACTION=fsop ARGS=mkdir -p /var/log/kafka/backups
[2026-02-05 03:49:52 UTC] USER=www-data EUID=0 PID=3606213 ACTION=fsop ARGS=chown -R kafka:kafka /var/backups/kafka/web-universe-main-dev
[2026-02-05 03:49:52 UTC] USER=www-data EUID=0 PID=3606223 ACTION=fsop ARGS=chown -R kafka:kafka /var/log/kafka/backups
[2026-02-05 03:49:52 UTC] USER=www-data EUID=0 PID=3606232 ACTION=fsop ARGS=chmod 750 /var/backups/kafka/web-universe-main-dev
[0;32m[OK][0m ✅ Local backup directory created
[0;34m[INFO][0m 3️⃣ Creating topic backup script...
[2026-02-05 03:49:53 UTC] USER=www-data EUID=0 PID=3606250 ACTION=fsop ARGS=sed -i s|__ENV_ID__|web-universe-main-dev|g /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[2026-02-05 03:49:53 UTC] USER=www-data EUID=0 PID=3606263 ACTION=fsop ARGS=sed -i s|__KAFKA_BROKER__|eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com:9092|g /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[2026-02-05 03:49:53 UTC] USER=www-data EUID=0 PID=3606272 ACTION=fsop ARGS=sed -i s|__BACKUP_DIR__|/var/backups/kafka/web-universe-main-dev|g /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[2026-02-05 03:49:53 UTC] USER=www-data EUID=0 PID=3606281 ACTION=fsop ARGS=sed -i s|__S3_BUCKET__|fastorder-kafka-backups-web-universe-main-dev|g /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[2026-02-05 03:49:53 UTC] USER=www-data EUID=0 PID=3606292 ACTION=fsop ARGS=sed -i s|__S3_REGION__|me-central-1|g /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[2026-02-05 03:49:53 UTC] USER=www-data EUID=0 PID=3606311 ACTION=fsop ARGS=chown root:kafka /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[0;32m[OK][0m ✅ Backup script created: /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[0;34m[INFO][0m 4️⃣ Setting up cron jobs for automated backups...
[2026-02-05 03:49:53 UTC] USER=www-data EUID=0 PID=3606331 ACTION=fsop ARGS=chmod 644 /etc/cron.d/kafka-backups-web-universe-main-dev
[0;32m[OK][0m ✅ Cron job configured: Daily backups at 2:00 AM
[0;34m[INFO][0m 5️⃣ Creating restore documentation...
[2026-02-05 03:49:53 UTC] USER=www-data EUID=0 PID=3606354 ACTION=fsop ARGS=sed -i s|__S3_BUCKET__|fastorder-kafka-backups-web-universe-main-dev|g /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[2026-02-05 03:49:53 UTC] USER=www-data EUID=0 PID=3606363 ACTION=fsop ARGS=sed -i s|__S3_REGION__|me-central-1|g /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[2026-02-05 03:49:53 UTC] USER=www-data EUID=0 PID=3606372 ACTION=fsop ARGS=sed -i s|__KAFKA_BROKER__|eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com|g /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[2026-02-05 03:49:53 UTC] USER=www-data EUID=0 PID=3606389 ACTION=fsop ARGS=chmod 644 /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[2026-02-05 03:49:53 UTC] USER=www-data EUID=0 PID=3606406 ACTION=fsop ARGS=chown kafka:kafka /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[0;32m[OK][0m ✅ Restore documentation created: /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Kafka Backup Configured
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m S3 Bucket: fastorder-kafka-backups-web-universe-main-dev
[0;34m[INFO][0m Region: me-central-1
[0;34m[INFO][0m Local backup dir: /var/backups/kafka/web-universe-main-dev
[0;34m[INFO][0m Schedule: Daily at 2:00 AM
[0;34m[INFO][0m Script: /usr/local/bin/kafka-backup-web-universe-main-dev.sh
[0;34m[INFO][0m Restore docs: /var/backups/kafka/web-universe-main-dev/RESTORE_INSTRUCTIONS.md
[1;33m[WARN][0m ⚠️ Note: This backs up Kafka metadata only (topics, configs, offsets)
[1;33m[WARN][0m For full message data backup, configure Kafka Connect S3 Sink
[0;32m[OK][0m ✅ Step 7 completed: 06-setup-backups.sh
[0;34m[INFO][0m 📦 Step 8/13: monitoring setup...
[INFO] Loaded environment: web-universe-main-dev (svc=web zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Kafka Monitoring Integration for web-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-web-universe-main-dev
[1;32m[OK][0m Observability cell endpoints registered for web-universe-main-dev
[1;32m[OK][0m ✓ Observability cell is ready
[INFO] 2️⃣ Setting up Kafka JMX exporter integration...
[INFO] JMX Exporter port calculated for web-universe-main-dev: 9362 (offset: 54)
[INFO] Checking observability cell readiness: obs-web-universe-main-dev
[1;32m[OK][0m Observability cell endpoints registered for web-universe-main-dev
[INFO] Setting up Kafka JMX exporter for web-universe-main-dev
[INFO] JMX Prometheus Java Agent already exists at /opt/kafka/libs/jmx_prometheus_javaagent.jar
[2026-02-05 03:49:53 UTC] USER=www-data EUID=0 PID=3606454 ACTION=passthru ARGS=mv /tmp/jmx_exporter.yml /opt/kafka/config/jmx_exporter.yml
[2026-02-05 03:49:54 UTC] USER=www-data EUID=0 PID=3606464 ACTION=passthru ARGS=chmod 644 /opt/kafka/config/jmx_exporter.yml
[1;32m[OK][0m JMX exporter configuration created at /opt/kafka/config/jmx_exporter.yml
[1;32m[OK][0m JMX exporter configuration created
[INFO] Configuring Kafka systemd services to use JMX exporter...
[2026-02-05 03:49:54 UTC] USER=www-data EUID=0 PID=3606495 ACTION=fsop ARGS=test -f /etc/systemd/system/[2026-02-05
[INFO] All Kafka services already configured with JMX exporter
[1;32m[OK][0m Kafka JMX exporter integration complete
[INFO] Metrics endpoint: http://142.93.238.16:9362/metrics
[INFO] Prometheus will automatically scrape: https://metrics-web-universe-main-dev.fastorder.com:9090
[INFO] View dashboards at: https://dashboards-web-universe-main-dev.fastorder.com
[1;32m[OK][0m ✓ Kafka JMX exporter integration complete
[INFO] Configuring KAFKA_OPTS environment variable for kafka user...
[2026-02-05 03:49:54 UTC] USER=www-data EUID=0 PID=3606517 ACTION=passthru ARGS=grep -q KAFKA_OPTS.*javaagent.*jmx_prometheus_javaagent.*=9362: /home/kafka/.bashrc
[2026-02-05 03:49:54 UTC] USER=www-data EUID=0 PID=3606538 ACTION=passthru ARGS=sed -i /export KAFKA_OPTS=.*jmx_prometheus_javaagent/d /home/kafka/.bashrc
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[1;32m[OK][0m ✓ KAFKA_OPTS configured in /home/kafka/.bashrc (port 9362)
[INFO] 2.5️⃣ Enabling JMX exporter in Kafka systemd service...
[2026-02-05 03:49:54 UTC] USER=www-data EUID=0 PID=3606572 ACTION=passthru ARGS=grep -q javaagent.*jmx_prometheus_javaagent /etc/systemd/system/confluent-kraft-web-universe-main-dev_coordinator.service
[INFO] Updating confluent-kraft-web-universe-main-dev_coordinator.service to enable JMX exporter...
[2026-02-05 03:49:54 UTC] USER=www-data EUID=0 PID=3606593 ACTION=passthru ARGS=sed -i s|^Environment=KAFKA_OPTS=.*|Environment=KAFKA_OPTS=-javaagent:/opt/kafka/libs/jmx_prometheus_javaagent.jar=9362:/opt/kafka/config/jmx_exporter.yml| /etc/systemd/system/confluent-kraft-web-universe-main-dev_coordinator.service
[1;32m[OK][0m ✓ Updated confluent-kraft-web-universe-main-dev_coordinator.service
[INFO] Reloading systemd daemon and restarting Kafka services...
[2026-02-05 03:49:54 UTC] USER=www-data EUID=0 PID=3606615 ACTION=passthru ARGS=systemctl daemon-reload
[2026-02-05 03:49:54 UTC] USER=www-data EUID=0 PID=3606674 ACTION=passthru ARGS=systemctl is-active --quiet confluent-kraft-web-universe-main-dev_coordinator
[INFO] Restarting confluent-kraft-web-universe-main-dev_coordinator...
[2026-02-05 03:49:55 UTC] USER=www-data EUID=0 PID=3606697 ACTION=passthru ARGS=systemctl restart confluent-kraft-web-universe-main-dev_coordinator
[2026-02-05 03:49:59 UTC] USER=www-data EUID=0 PID=3607331 ACTION=passthru ARGS=systemctl is-active --quiet confluent-kraft-web-universe-main-dev_coordinator
[1;32m[OK][0m ✓ confluent-kraft-web-universe-main-dev_coordinator restarted successfully
[1;32m[OK][0m ✓ JMX exporter enabled in Kafka systemd services
[INFO] 2.6️⃣ Configuring Prometheus to scrape Kafka metrics...
[2026-02-05 03:49:59 UTC] USER=www-data EUID=0 PID=3607353 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
[INFO] Adding Kafka scrape target to Prometheus configuration...
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[2026-02-05 03:49:59 UTC] USER=www-data EUID=0 PID=3607386 ACTION=passthru ARGS=sed -i /# Prometheus self-monitoring/r /tmp/prometheus_kafka_add.yml /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
[ERROR] Invalid Prometheus configuration - rolling back
[2026-02-05 03:49:59 UTC] USER=www-data EUID=0 PID=3607419 ACTION=passthru ARGS=sed -i /job_name: 'kafka'/,+6d /etc/prometheus/obs-web-universe-main-dev/prometheus.yml
[2026-02-05 03:49:59 UTC] USER=www-data EUID=0 PID=3607444 ACTION=fsop ARGS=rm -f /tmp/prometheus_kafka_add.yml
[INFO] 3️⃣ Registering Kafka nodes to monitoring database...
[INFO] Detected Kafka version: 3.9.1
[INFO] Registering Kafka Broker to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Kafka Broker
[INFO] Identifier: web-universe-main-dev-broker-01
[INFO] Identifier Parent: cluster
[INFO] IP: 142.93.238.16
[INFO] Port: 9092
[INFO] FQDN: eventbus-web-universe-main-dev-kafka-broker-01.fastorder.com
[INFO] Status: running
[INFO] Environment: web-universe-main-dev (service=web, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[WARN] Registration API call failed (HTTP 500), retrying (1/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[WARN] Registration API call failed (HTTP 500), retrying (2/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ REGISTRATION FAILED AFTER 3 ATTEMPTS
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] HTTP Status: 500
[ERROR] Response: {"success":false,"error":"Registration failed: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation \"obs.application\" does not exist\nLINE 2: SELECT uuid FROM obs.application\n ^"}
[ERROR]
[ERROR] API endpoint: https://skeleton.dev.fastorder.com/api/obs/register
[ERROR]
[ERROR] Troubleshooting:
[ERROR] 1. Check if skeleton.dev.fastorder.com is accessible
[ERROR] 2. Verify web application is running
[ERROR] 3. Check web application logs: /var/www/html/skeleton.dev.fastorder.com/logs/
[ERROR] 4. Test API manually:
[ERROR] curl -k -X POST 'https://skeleton.dev.fastorder.com/api/obs/register' \
[ERROR] -H 'Content-Type: application/json' \
[ERROR] -H 'X-Internal-Token: $OBS_INTERNAL_API_TOKEN' \
[ERROR] -d '$PAYLOAD'
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ KAFKA BROKER REGISTRATION FAILED
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] Failed to register Kafka Broker to observability dashboard
[ERROR]
[ERROR] Check API token: sudo cat /etc/fastorder/obs-api-token
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;31m[ERROR][0m ❌ Step 8 failed: 10-monitoring-setup.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;31m[ERROR][0m Kafka setup FAILED at step: 10-monitoring-setup.sh
[0;31m[ERROR][0m Executed 8 of 13 steps
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;31m[ERROR][0m ❌ Event bus infrastructure (kafka) setup failed with exit code: 1
⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...